2024-09-13T09:12:34.581 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T09:12:34.632 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T09:12:34.720 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7901034 branch: reef description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} email: ceph-qa@ceph.com first_in_suite: false job_id: '7901034' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: reef ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 48a7de7e6e1701a88fd571c7e34908ac4b8ed412 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 48a7de7e6e1701a88fd571c7e34908ac4b8ed412 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: 7a853b1706d076d58e0da7a5d3b9b9eb36276880 owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 248 sha1: 48a7de7e6e1701a88fd571c7e34908ac4b8ed412 sleep_before_teardown: 0 subset: 116/128 suite: rbd suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_7a853b1706d076d58e0da7a5d3b9b9eb36276880/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 7a853b1706d076d58e0da7a5d3b9b9eb36276880 targets: smithi122.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBPLTJok4PZJYDU2rwScvJ7rB23VRq1g0ehgis1fGIuL1AVwW47oni1FTJmM56obzAHWh8Hf7+Ni6NVwEMRUVuA= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD/2hn0Bf/KzJKwZT28wcQLhGlXAxiKjAtWMHBFSyYnSZxSL3Z81UbS2xmNOY2cVn5onuYGXxBkdym2Qn+M0/v8= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_IMAGE_MODE: snapshot MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_22:16:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T09:12:34.721 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_7a853b1706d076d58e0da7a5d3b9b9eb36276880/qa; will attempt to use it 2024-09-13T09:12:34.722 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_7a853b1706d076d58e0da7a5d3b9b9eb36276880/qa/tasks 2024-09-13T09:12:34.722 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T09:12:34.723 INFO:teuthology.task.internal:Checking packages... 2024-09-13T09:12:34.744 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '48a7de7e6e1701a88fd571c7e34908ac4b8ed412' 2024-09-13T09:12:34.744 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T09:12:34.744 INFO:teuthology.packaging:ref: None 2024-09-13T09:12:34.744 INFO:teuthology.packaging:tag: None 2024-09-13T09:12:34.744 INFO:teuthology.packaging:branch: reef 2024-09-13T09:12:34.744 INFO:teuthology.packaging:sha1: 48a7de7e6e1701a88fd571c7e34908ac4b8ed412 2024-09-13T09:12:34.744 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-13T09:12:34.974 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-843.g2c01458a 2024-09-13T09:12:34.976 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T09:12:34.988 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T09:12:34.988 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T09:12:35.043 INFO:teuthology.task.internal:Saving configuration 2024-09-13T09:12:35.059 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T09:12:35.076 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T09:12:35.100 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7901034', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 09:07:39.845778', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBPLTJok4PZJYDU2rwScvJ7rB23VRq1g0ehgis1fGIuL1AVwW47oni1FTJmM56obzAHWh8Hf7+Ni6NVwEMRUVuA='} 2024-09-13T09:12:35.125 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7901034', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 09:07:39.844291', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD/2hn0Bf/KzJKwZT28wcQLhGlXAxiKjAtWMHBFSyYnSZxSL3Z81UbS2xmNOY2cVn5onuYGXxBkdym2Qn+M0/v8='} 2024-09-13T09:12:35.125 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T09:12:35.136 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-09-13T09:12:35.136 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-09-13T09:12:35.137 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T09:12:35.221 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8caf93bbe0>, signals=[15]) 2024-09-13T09:12:35.221 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T09:12:35.232 INFO:teuthology.task.internal:Opening connections... 2024-09-13T09:12:35.232 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T09:12:35.233 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T09:12:35.307 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2024-09-13T09:12:35.307 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T09:12:35.380 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T09:12:35.389 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2024-09-13T09:12:35.407 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2024-09-13T09:12:35.408 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2024-09-13T09:12:35.465 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="CentOS Stream" 2024-09-13T09:12:35.465 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="9" 2024-09-13T09:12:35.465 INFO:teuthology.orchestra.run.smithi122.stdout:ID="centos" 2024-09-13T09:12:35.465 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="rhel fedora" 2024-09-13T09:12:35.465 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="9" 2024-09-13T09:12:35.465 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el9" 2024-09-13T09:12:35.465 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T09:12:35.465 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2024-09-13T09:12:35.465 INFO:teuthology.orchestra.run.smithi122.stdout:LOGO="fedora-logo-icon" 2024-09-13T09:12:35.466 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T09:12:35.466 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://centos.org/" 2024-09-13T09:12:35.466 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T09:12:35.466 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T09:12:35.466 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T09:12:35.466 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2024-09-13T09:12:35.493 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2024-09-13T09:12:35.512 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2024-09-13T09:12:35.512 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2024-09-13T09:12:35.571 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="CentOS Stream" 2024-09-13T09:12:35.571 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="9" 2024-09-13T09:12:35.571 INFO:teuthology.orchestra.run.smithi190.stdout:ID="centos" 2024-09-13T09:12:35.571 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE="rhel fedora" 2024-09-13T09:12:35.571 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="9" 2024-09-13T09:12:35.571 INFO:teuthology.orchestra.run.smithi190.stdout:PLATFORM_ID="platform:el9" 2024-09-13T09:12:35.571 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T09:12:35.571 INFO:teuthology.orchestra.run.smithi190.stdout:ANSI_COLOR="0;31" 2024-09-13T09:12:35.571 INFO:teuthology.orchestra.run.smithi190.stdout:LOGO="fedora-logo-icon" 2024-09-13T09:12:35.572 INFO:teuthology.orchestra.run.smithi190.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T09:12:35.572 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://centos.org/" 2024-09-13T09:12:35.572 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T09:12:35.572 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T09:12:35.572 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T09:12:35.572 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2024-09-13T09:12:35.602 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T09:12:35.613 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T09:12:35.641 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T09:12:35.642 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2024-09-13T09:12:35.644 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2024-09-13T09:12:35.661 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T09:12:35.675 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T09:12:35.675 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2024-09-13T09:12:35.701 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2024-09-13T09:12:35.724 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T09:12:35.782 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T09:12:35.801 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-13T09:12:35.801 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-13T09:12:35.801 DEBUG:teuthology.orchestra.run.smithi122:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T09:12:35.802 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T09:12:35.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T09:12:35.818 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-09-13T09:12:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T09:12:35.819 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-09-13T09:12:35.876 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-503.el9.x86_64 2024-09-13T09:12:35.876 INFO:teuthology.task.kernel:Running kernel on smithi122: 5.14.0-503.el9.x86_64 2024-09-13T09:12:35.877 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum install -y kernel 2024-09-13T09:12:35.878 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-503.el9.x86_64 2024-09-13T09:12:35.878 INFO:teuthology.task.kernel:Running kernel on smithi190: 5.14.0-503.el9.x86_64 2024-09-13T09:12:35.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum install -y kernel 2024-09-13T09:12:39.573 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.2 MB 00:01 2024-09-13T09:12:40.259 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 3.1 MB/s | 8.2 MB 00:02 2024-09-13T09:12:41.366 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 29 MB/s | 20 MB 00:00 2024-09-13T09:12:43.661 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 8.9 MB/s | 20 MB 00:02 2024-09-13T09:12:47.184 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.5 MB 00:02 2024-09-13T09:12:48.635 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 19 kB 00:00 2024-09-13T09:12:49.989 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 18 MB/s | 23 MB 00:01 2024-09-13T09:12:50.664 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 2.0 MB/s | 6.5 MB 00:03 2024-09-13T09:12:52.311 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 19 kB 00:00 2024-09-13T09:12:53.149 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 32 MB/s | 23 MB 00:00 2024-09-13T09:12:55.487 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-13T09:12:57.476 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T09:12:57.477 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T09:12:57.567 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:12:57.578 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:12:57.578 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-09-13T09:12:57.579 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:12:57.579 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T09:12:57.579 INFO:teuthology.orchestra.run.smithi122.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T09:12:57.579 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T09:12:57.579 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T09:12:57.579 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T09:12:57.579 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T09:12:57.579 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:12:57.579 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T09:12:57.579 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:12:57.579 INFO:teuthology.orchestra.run.smithi122.stdout:Install 4 Packages 2024-09-13T09:12:57.579 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:12:57.580 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 79 M 2024-09-13T09:12:57.580 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 126 M 2024-09-13T09:12:57.580 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T09:12:57.688 INFO:teuthology.orchestra.run.smithi122.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 260 kB/s | 23 kB 00:00 2024-09-13T09:12:58.679 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-09-13T09:13:00.661 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T09:13:00.661 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T09:13:00.750 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:13:00.761 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:13:00.761 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-09-13T09:13:00.761 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:13:00.761 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-13T09:13:00.761 INFO:teuthology.orchestra.run.smithi190.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T09:13:00.761 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-13T09:13:00.762 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T09:13:00.762 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T09:13:00.762 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T09:13:00.762 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:13:00.762 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-13T09:13:00.762 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:13:00.762 INFO:teuthology.orchestra.run.smithi190.stdout:Install 4 Packages 2024-09-13T09:13:00.762 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:13:00.763 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 79 M 2024-09-13T09:13:00.763 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 126 M 2024-09-13T09:13:00.763 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-13T09:13:00.880 INFO:teuthology.orchestra.run.smithi122.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 4.8 MB/s | 16 MB 00:03 2024-09-13T09:13:01.339 INFO:teuthology.orchestra.run.smithi190.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 77 kB/s | 23 kB 00:00 2024-09-13T09:13:03.338 INFO:teuthology.orchestra.run.smithi122.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 5.1 MB/s | 29 MB 00:05 2024-09-13T09:13:04.014 INFO:teuthology.orchestra.run.smithi122.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 5.4 MB/s | 35 MB 00:06 2024-09-13T09:13:04.014 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:13:04.014 INFO:teuthology.orchestra.run.smithi122.stdout:Total 12 MB/s | 79 MB 00:06 2024-09-13T09:13:04.697 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T09:13:05.035 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T09:13:05.035 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T09:13:06.009 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T09:13:06.009 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T09:13:08.452 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T09:13:09.191 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T09:13:09.276 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T09:13:12.073 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T09:13:12.202 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T09:13:13.313 INFO:teuthology.orchestra.run.smithi190.stdout:(2/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 2.8 MB/s | 35 MB 00:12 2024-09-13T09:13:17.061 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T09:13:17.171 INFO:teuthology.orchestra.run.smithi190.stdout:(3/4): kernel-core-5.14.0-505.el9.x86_64.rpm 993 kB/s | 16 MB 00:16 2024-09-13T09:13:17.219 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T09:13:22.067 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T09:13:32.320 INFO:teuthology.orchestra.run.smithi190.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 944 kB/s | 29 MB 00:30 2024-09-13T09:13:32.320 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:13:32.320 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.5 MB/s | 79 MB 00:31 2024-09-13T09:13:33.006 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-13T09:13:33.357 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-13T09:13:33.357 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-13T09:13:34.318 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-13T09:13:34.319 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-13T09:13:36.765 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-13T09:13:37.404 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T09:13:37.486 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T09:13:40.261 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T09:13:40.389 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T09:13:45.203 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T09:13:45.240 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T09:13:50.095 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T09:14:00.564 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T09:14:00.604 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T09:14:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T09:14:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T09:14:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T09:14:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T09:14:01.816 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T09:14:01.816 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:14:01.817 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T09:14:01.817 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T09:14:01.817 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T09:14:01.817 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T09:14:01.817 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T09:14:01.817 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:14:01.817 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:14:02.181 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T09:14:02.266 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T09:14:02.266 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T09:14:02.267 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T09:14:02.267 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T09:14:02.267 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T09:14:02.267 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-09-13T09:14:02.267 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi122.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T09:14:02.267 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T09:14:02.325 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T09:14:02.326 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T09:14:02.326 DEBUG:teuthology.orchestra.run.smithi122:> sudo rpm -qi grub2-tools 2024-09-13T09:14:02.363 INFO:teuthology.orchestra.run.smithi122.stdout:Name : grub2-tools 2024-09-13T09:14:02.363 INFO:teuthology.orchestra.run.smithi122.stdout:Epoch : 1 2024-09-13T09:14:02.363 INFO:teuthology.orchestra.run.smithi122.stdout:Version : 2.06 2024-09-13T09:14:02.363 INFO:teuthology.orchestra.run.smithi122.stdout:Release : 82.el9 2024-09-13T09:14:02.363 INFO:teuthology.orchestra.run.smithi122.stdout:Architecture: x86_64 2024-09-13T09:14:02.363 INFO:teuthology.orchestra.run.smithi122.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T09:14:02.363 INFO:teuthology.orchestra.run.smithi122.stdout:Group : Unspecified 2024-09-13T09:14:02.363 INFO:teuthology.orchestra.run.smithi122.stdout:Size : 8274670 2024-09-13T09:14:02.363 INFO:teuthology.orchestra.run.smithi122.stdout:License : GPLv3+ 2024-09-13T09:14:02.364 INFO:teuthology.orchestra.run.smithi122.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T09:14:02.364 INFO:teuthology.orchestra.run.smithi122.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T09:14:02.364 INFO:teuthology.orchestra.run.smithi122.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T09:14:02.364 INFO:teuthology.orchestra.run.smithi122.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T09:14:02.364 INFO:teuthology.orchestra.run.smithi122.stdout:Packager : builder@centos.org 2024-09-13T09:14:02.364 INFO:teuthology.orchestra.run.smithi122.stdout:Vendor : CentOS 2024-09-13T09:14:02.364 INFO:teuthology.orchestra.run.smithi122.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T09:14:02.364 INFO:teuthology.orchestra.run.smithi122.stdout:Summary : Support tools for GRUB. 2024-09-13T09:14:02.364 INFO:teuthology.orchestra.run.smithi122.stdout:Description : 2024-09-13T09:14:02.364 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:14:02.364 INFO:teuthology.orchestra.run.smithi122.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T09:14:02.364 INFO:teuthology.orchestra.run.smithi122.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T09:14:02.364 INFO:teuthology.orchestra.run.smithi122.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T09:14:02.365 INFO:teuthology.orchestra.run.smithi122.stdout:hardware devices. 2024-09-13T09:14:02.365 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:14:02.365 INFO:teuthology.orchestra.run.smithi122.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T09:14:02.365 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T09:14:02.366 INFO:teuthology.task.kernel:Updating grub on smithi122 to boot 5.14.0-505.el9.x86_64 2024-09-13T09:14:02.366 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T09:14:03.029 INFO:teuthology.orchestra.run.smithi122.stderr:Generating grub configuration file ... 2024-09-13T09:14:04.334 INFO:teuthology.orchestra.run.smithi122.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T09:14:04.364 INFO:teuthology.orchestra.run.smithi122.stderr:done 2024-09-13T09:14:04.366 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-09-13T09:14:04.381 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.8oHNT8mgMu 2024-09-13T09:14:04.382 DEBUG:teuthology.orchestra.run.smithi122:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.8oHNT8mgMu 2024-09-13T09:14:04.449 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.8oHNT8mgMu 2024-09-13T09:14:04.587 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.8oHNT8mgMu is 6KB 2024-09-13T09:14:04.636 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.8oHNT8mgMu 2024-09-13T09:14:04.651 DEBUG:teuthology.orchestra.run.smithi122:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T09:14:04.713 INFO:teuthology.orchestra.run.smithi122.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T09:14:04.714 INFO:teuthology.orchestra.run.smithi122.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T09:14:04.714 INFO:teuthology.orchestra.run.smithi122.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T09:14:04.714 INFO:teuthology.orchestra.run.smithi122.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T09:14:04.715 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T09:14:04.879 DEBUG:teuthology.orchestra.run.smithi122:> sudo shutdown -r now 2024-09-13T09:14:28.567 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T09:14:28.615 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T09:14:29.015 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T09:14:29.015 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T09:14:29.015 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T09:14:29.016 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T09:14:29.845 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T09:14:29.845 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:14:29.846 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-13T09:14:29.846 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T09:14:29.846 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T09:14:29.846 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T09:14:29.846 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T09:14:29.846 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:14:29.846 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:14:30.187 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T09:14:30.275 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T09:14:30.275 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T09:14:30.275 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T09:14:30.275 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T09:14:30.275 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T09:14:30.276 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-09-13T09:14:30.276 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi190.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T09:14:30.276 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T09:14:30.335 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T09:14:30.336 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T09:14:30.336 DEBUG:teuthology.orchestra.run.smithi190:> sudo rpm -qi grub2-tools 2024-09-13T09:14:30.368 INFO:teuthology.orchestra.run.smithi190.stdout:Name : grub2-tools 2024-09-13T09:14:30.368 INFO:teuthology.orchestra.run.smithi190.stdout:Epoch : 1 2024-09-13T09:14:30.368 INFO:teuthology.orchestra.run.smithi190.stdout:Version : 2.06 2024-09-13T09:14:30.369 INFO:teuthology.orchestra.run.smithi190.stdout:Release : 82.el9 2024-09-13T09:14:30.369 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: x86_64 2024-09-13T09:14:30.369 INFO:teuthology.orchestra.run.smithi190.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T09:14:30.369 INFO:teuthology.orchestra.run.smithi190.stdout:Group : Unspecified 2024-09-13T09:14:30.369 INFO:teuthology.orchestra.run.smithi190.stdout:Size : 8274670 2024-09-13T09:14:30.369 INFO:teuthology.orchestra.run.smithi190.stdout:License : GPLv3+ 2024-09-13T09:14:30.369 INFO:teuthology.orchestra.run.smithi190.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T09:14:30.369 INFO:teuthology.orchestra.run.smithi190.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T09:14:30.369 INFO:teuthology.orchestra.run.smithi190.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T09:14:30.369 INFO:teuthology.orchestra.run.smithi190.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T09:14:30.369 INFO:teuthology.orchestra.run.smithi190.stdout:Packager : builder@centos.org 2024-09-13T09:14:30.369 INFO:teuthology.orchestra.run.smithi190.stdout:Vendor : CentOS 2024-09-13T09:14:30.370 INFO:teuthology.orchestra.run.smithi190.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T09:14:30.370 INFO:teuthology.orchestra.run.smithi190.stdout:Summary : Support tools for GRUB. 2024-09-13T09:14:30.370 INFO:teuthology.orchestra.run.smithi190.stdout:Description : 2024-09-13T09:14:30.370 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:14:30.370 INFO:teuthology.orchestra.run.smithi190.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T09:14:30.370 INFO:teuthology.orchestra.run.smithi190.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T09:14:30.370 INFO:teuthology.orchestra.run.smithi190.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T09:14:30.370 INFO:teuthology.orchestra.run.smithi190.stdout:hardware devices. 2024-09-13T09:14:30.370 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:14:30.370 INFO:teuthology.orchestra.run.smithi190.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T09:14:30.371 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T09:14:30.371 INFO:teuthology.task.kernel:Updating grub on smithi190 to boot 5.14.0-505.el9.x86_64 2024-09-13T09:14:30.372 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T09:14:31.030 INFO:teuthology.orchestra.run.smithi190.stderr:Generating grub configuration file ... 2024-09-13T09:14:32.380 INFO:teuthology.orchestra.run.smithi190.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T09:14:32.409 INFO:teuthology.orchestra.run.smithi190.stderr:done 2024-09-13T09:14:32.411 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-09-13T09:14:32.430 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.R2leZaYNkw 2024-09-13T09:14:32.430 DEBUG:teuthology.orchestra.run.smithi190:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.R2leZaYNkw 2024-09-13T09:14:32.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.R2leZaYNkw 2024-09-13T09:14:32.649 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.R2leZaYNkw is 6KB 2024-09-13T09:14:32.698 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.R2leZaYNkw 2024-09-13T09:14:32.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T09:14:32.781 INFO:teuthology.orchestra.run.smithi190.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T09:14:32.781 INFO:teuthology.orchestra.run.smithi190.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T09:14:32.782 INFO:teuthology.orchestra.run.smithi190.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T09:14:32.782 INFO:teuthology.orchestra.run.smithi190.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T09:14:32.783 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T09:14:32.930 DEBUG:teuthology.orchestra.run.smithi190:> sudo shutdown -r now 2024-09-13T09:14:34.884 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-09-13T09:14:34.884 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T09:14:34.885 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T09:14:53.377 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-09-13T09:15:02.385 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T09:15:02.386 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T09:15:02.933 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-09-13T09:15:02.933 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-13T09:15:02.934 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T09:15:05.437 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-09-13T09:15:17.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T09:15:17.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T09:15:21.277 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-09-13T09:15:30.285 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-13T09:15:30.286 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T09:15:49.300 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-09-13T09:15:49.806 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T09:15:49.806 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-13T09:15:49.806 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-09-13T09:15:49.820 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-505.el9.x86_64 2024-09-13T09:15:49.821 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T09:15:49.821 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T09:15:49.821 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T09:15:50.821 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-09-13T09:15:50.822 DEBUG:teuthology.orchestra.run.smithi122:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T09:15:50.931 INFO:teuthology.orchestra.run.smithi122.stdout:ttyS1 2024-09-13T09:15:50.968 DEBUG:teuthology.parallel:result is None 2024-09-13T09:16:30.289 DEBUG:teuthology.orchestra.remote:timed out 2024-09-13T09:16:42.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-13T09:16:42.292 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T09:16:42.677 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-09-13T09:16:43.016 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-13T09:16:43.017 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-13T09:16:43.017 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-09-13T09:16:43.033 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-505.el9.x86_64 2024-09-13T09:16:43.033 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T09:16:43.033 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T09:16:43.034 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T09:16:44.035 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-09-13T09:16:44.035 DEBUG:teuthology.orchestra.run.smithi190:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T09:16:44.102 INFO:teuthology.orchestra.run.smithi190.stdout:ttyS1 2024-09-13T09:16:44.124 DEBUG:teuthology.parallel:result is None 2024-09-13T09:16:44.124 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T09:16:44.132 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T09:16:44.132 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T09:16:44.135 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T09:16:44.153 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T09:16:44.174 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T09:16:44.215 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T09:16:44.216 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T09:16:44.219 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T09:16:44.280 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T09:16:44.286 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T09:16:44.286 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T09:16:44.288 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T09:16:44.325 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T09:16:44.326 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T09:16:44.337 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T09:16:44.337 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T09:16:44.339 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T09:16:44.345 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T09:16:44.345 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T09:16:44.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T09:16:44.409 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T09:16:44.418 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T09:16:44.418 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T09:16:44.453 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T09:16:44.469 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T09:16:44.555 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T09:16:44.613 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T09:16:44.699 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T09:16:44.745 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:16:44.745 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T09:16:44.814 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T09:16:44.858 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T09:16:44.904 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T09:16:44.990 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T09:16:45.029 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:16:45.029 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T09:16:45.095 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2024-09-13T09:16:45.097 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2024-09-13T09:16:45.145 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T09:16:45.185 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T09:16:45.658 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T09:16:45.665 INFO:teuthology.task.internal:Starting timer... 2024-09-13T09:16:45.665 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T09:16:45.711 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T09:16:45.752 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-13T09:16:45.752 DEBUG:teuthology.orchestra.run.smithi122:> sudo service auditd rotate 2024-09-13T09:16:45.828 INFO:teuthology.orchestra.run.smithi122.stdout:Rotating logs: 2024-09-13T09:16:45.830 DEBUG:teuthology.orchestra.run.smithi190:> sudo service auditd rotate 2024-09-13T09:16:45.896 INFO:teuthology.orchestra.run.smithi190.stdout:Rotating logs: 2024-09-13T09:16:45.900 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T09:16:45.900 DEBUG:teuthology.orchestra.run.smithi122:> /usr/sbin/getenforce 2024-09-13T09:16:45.929 INFO:teuthology.orchestra.run.smithi122.stdout:Permissive 2024-09-13T09:16:45.929 DEBUG:teuthology.orchestra.run.smithi190:> /usr/sbin/getenforce 2024-09-13T09:16:45.955 INFO:teuthology.orchestra.run.smithi190.stdout:Permissive 2024-09-13T09:16:45.955 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi122.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi190.front.sepia.ceph.com': 'permissive'} 2024-09-13T09:16:45.955 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-13T09:16:45.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T09:16:46.000 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-13T09:16:46.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T09:16:46.027 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T09:16:46.027 DEBUG:teuthology.orchestra.run.smithi122:> sudo /usr/sbin/setenforce permissive 2024-09-13T09:16:46.080 DEBUG:teuthology.orchestra.run.smithi190:> sudo /usr/sbin/setenforce permissive 2024-09-13T09:16:46.113 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T09:16:46.169 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T09:16:46.182 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T09:16:46.183 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi122.front.sepia.ceph.com,smithi190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T09:20:28.957 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi122.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com')] 2024-09-13T09:20:28.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T09:20:28.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T09:20:29.037 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-09-13T09:20:29.118 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T09:20:29.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-13T09:20:29.119 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T09:20:29.196 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-09-13T09:20:29.281 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-13T09:20:29.281 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T09:20:29.291 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T09:20:29.292 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T09:20:29.292 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T09:20:29.295 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T09:20:29.295 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T09:20:29.326 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T09:20:29.341 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T09:20:29.351 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T09:20:29.366 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T09:20:29.367 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: ntpd: command not found 2024-09-13T09:20:29.379 INFO:teuthology.orchestra.run.smithi122.stdout:506 Cannot talk to daemon 2024-09-13T09:20:29.390 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: ntpd: command not found 2024-09-13T09:20:29.394 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T09:20:29.402 INFO:teuthology.orchestra.run.smithi190.stdout:506 Cannot talk to daemon 2024-09-13T09:20:29.408 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T09:20:29.416 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T09:20:29.430 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T09:20:29.462 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-09-13T09:20:29.466 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T09:20:29.466 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-09-13T09:20:29.466 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T09:20:29.466 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T09:20:29.466 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T09:20:29.466 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T09:20:29.485 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-09-13T09:20:29.489 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T09:20:29.489 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-09-13T09:20:29.489 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T09:20:29.489 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T09:20:29.489 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T09:20:29.489 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T09:20:29.490 INFO:teuthology.run_tasks:Running task install... 2024-09-13T09:20:29.500 DEBUG:teuthology.task.install:project ceph 2024-09-13T09:20:29.500 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412'}} 2024-09-13T09:20:29.501 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412'} 2024-09-13T09:20:29.501 INFO:teuthology.task.install:Using flavor: default 2024-09-13T09:20:29.506 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T09:20:29.506 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-09-13T09:20:29.507 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412', 'tag': None, 'wait_for_package': False} 2024-09-13T09:20:29.507 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=48a7de7e6e1701a88fd571c7e34908ac4b8ed412 2024-09-13T09:20:29.510 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412', 'tag': None, 'wait_for_package': False} 2024-09-13T09:20:29.510 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=48a7de7e6e1701a88fd571c7e34908ac4b8ed412 2024-09-13T09:20:29.659 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/centos/9/flavors/default/ 2024-09-13T09:20:29.659 INFO:teuthology.task.install.rpm:Package version is 18.2.4-820.g48a7de7e 2024-09-13T09:20:29.663 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/centos/9/flavors/default/ 2024-09-13T09:20:29.663 INFO:teuthology.task.install.rpm:Package version is 18.2.4-820.g48a7de7e 2024-09-13T09:20:29.772 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T09:20:29.773 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:20:29.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T09:20:29.778 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T09:20:29.778 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:20:29.778 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T09:20:29.799 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-13T09:20:29.799 DEBUG:teuthology.orchestra.run.smithi190:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T09:20:29.804 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-13T09:20:29.804 DEBUG:teuthology.orchestra.run.smithi122:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T09:20:29.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T09:20:29.870 DEBUG:teuthology.orchestra.run.smithi122:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T09:20:29.941 DEBUG:teuthology.orchestra.run.smithi190:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T09:20:29.946 DEBUG:teuthology.orchestra.run.smithi122:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T09:20:30.005 INFO:teuthology.orchestra.run.smithi190.stdout:check_obsoletes = 1 2024-09-13T09:20:30.007 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum clean all 2024-09-13T09:20:30.010 INFO:teuthology.orchestra.run.smithi122.stdout:check_obsoletes = 1 2024-09-13T09:20:30.012 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum clean all 2024-09-13T09:20:30.235 INFO:teuthology.orchestra.run.smithi122.stdout:45 files removed 2024-09-13T09:20:30.243 INFO:teuthology.orchestra.run.smithi190.stdout:45 files removed 2024-09-13T09:20:30.262 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-radosgw 2024-09-13T09:20:30.270 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-radosgw 2024-09-13T09:20:30.810 INFO:teuthology.orchestra.run.smithi190.stdout:ceph packages for x86_64 302 kB/s | 77 kB 00:00 2024-09-13T09:20:30.812 INFO:teuthology.orchestra.run.smithi122.stdout:ceph packages for x86_64 290 kB/s | 77 kB 00:00 2024-09-13T09:20:31.095 INFO:teuthology.orchestra.run.smithi190.stdout:ceph noarch packages 48 kB/s | 11 kB 00:00 2024-09-13T09:20:31.096 INFO:teuthology.orchestra.run.smithi122.stdout:ceph noarch packages 48 kB/s | 11 kB 00:00 2024-09-13T09:20:31.355 INFO:teuthology.orchestra.run.smithi122.stdout:ceph source packages 8.4 kB/s | 1.9 kB 00:00 2024-09-13T09:20:31.370 INFO:teuthology.orchestra.run.smithi190.stdout:ceph source packages 7.8 kB/s | 1.9 kB 00:00 2024-09-13T09:20:31.895 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.2 MB 00:00 2024-09-13T09:20:32.005 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-13T09:20:33.605 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 34 MB/s | 20 MB 00:00 2024-09-13T09:20:39.102 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.5 MB 00:01 2024-09-13T09:20:40.674 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 79 kB/s | 19 kB 00:00 2024-09-13T09:20:40.864 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 20 MB 00:07 2024-09-13T09:20:41.461 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-13T09:20:46.565 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.5 MB 00:01 2024-09-13T09:20:46.951 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-13T09:20:48.416 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 19 kB 00:00 2024-09-13T09:20:49.245 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 32 MB/s | 23 MB 00:00 2024-09-13T09:20:49.335 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:20:49.347 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:20:49.347 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-13T09:20:49.347 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:20:49.347 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-13T09:20:49.347 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 7.7 M 2024-09-13T09:20:49.347 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrading: 2024-09-13T09:20:49.347 INFO:teuthology.orchestra.run.smithi190.stdout: librados2 x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 3.3 M 2024-09-13T09:20:49.347 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1 x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 3.0 M 2024-09-13T09:20:49.347 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-13T09:20:49.348 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T09:20:49.348 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 5.1 M 2024-09-13T09:20:49.348 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 18 M 2024-09-13T09:20:49.348 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 25 k 2024-09-13T09:20:49.348 INFO:teuthology.orchestra.run.smithi190.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-13T09:20:49.348 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T09:20:49.348 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T09:20:49.348 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T09:20:49.348 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2 x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 711 k 2024-09-13T09:20:49.348 INFO:teuthology.orchestra.run.smithi190.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T09:20:49.348 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T09:20:49.348 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T09:20:49.348 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1 x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 478 k 2024-09-13T09:20:49.348 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T09:20:49.349 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2 x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 4.5 M 2024-09-13T09:20:49.349 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T09:20:49.349 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T09:20:49.349 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T09:20:49.349 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 45 k 2024-09-13T09:20:49.349 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 129 k 2024-09-13T09:20:49.349 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 162 k 2024-09-13T09:20:49.349 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T09:20:49.349 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 321 k 2024-09-13T09:20:49.349 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 299 k 2024-09-13T09:20:49.349 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 99 k 2024-09-13T09:20:49.349 INFO:teuthology.orchestra.run.smithi190.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T09:20:49.349 INFO:teuthology.orchestra.run.smithi190.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T09:20:49.350 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:20:49.350 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-13T09:20:49.350 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:20:49.350 INFO:teuthology.orchestra.run.smithi190.stdout:Install 28 Packages 2024-09-13T09:20:49.350 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrade 2 Packages 2024-09-13T09:20:49.350 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:20:49.350 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 53 M 2024-09-13T09:20:49.350 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-13T09:20:49.918 INFO:teuthology.orchestra.run.smithi190.stdout:(1/30): ceph-base-18.2.4-820.g48a7de7e.el9.x86_ 11 MB/s | 5.1 MB 00:00 2024-09-13T09:20:49.969 INFO:teuthology.orchestra.run.smithi190.stdout:(2/30): ceph-selinux-18.2.4-820.g48a7de7e.el9.x 504 kB/s | 25 kB 00:00 2024-09-13T09:20:50.111 INFO:teuthology.orchestra.run.smithi190.stdout:(3/30): ceph-radosgw-18.2.4-820.g48a7de7e.el9.x 12 MB/s | 7.7 MB 00:00 2024-09-13T09:20:50.144 INFO:teuthology.orchestra.run.smithi190.stdout:(4/30): libcephfs2-18.2.4-820.g48a7de7e.el9.x86 4.0 MB/s | 711 kB 00:00 2024-09-13T09:20:50.178 INFO:teuthology.orchestra.run.smithi190.stdout:(5/30): libradosstriper1-18.2.4-820.g48a7de7e.e 7.0 MB/s | 478 kB 00:00 2024-09-13T09:20:50.245 INFO:teuthology.orchestra.run.smithi190.stdout:(6/30): python3-ceph-argparse-18.2.4-820.g48a7d 676 kB/s | 45 kB 00:00 2024-09-13T09:20:50.312 INFO:teuthology.orchestra.run.smithi190.stdout:(7/30): python3-ceph-common-18.2.4-820.g48a7de7 1.9 MB/s | 129 kB 00:00 2024-09-13T09:20:50.379 INFO:teuthology.orchestra.run.smithi190.stdout:(8/30): librgw2-18.2.4-820.g48a7de7e.el9.x86_64 19 MB/s | 4.5 MB 00:00 2024-09-13T09:20:50.404 INFO:teuthology.orchestra.run.smithi190.stdout:(9/30): python3-cephfs-18.2.4-820.g48a7de7e.el9 1.7 MB/s | 162 kB 00:00 2024-09-13T09:20:50.438 INFO:teuthology.orchestra.run.smithi190.stdout:(10/30): python3-rados-18.2.4-820.g48a7de7e.el9 5.4 MB/s | 321 kB 00:00 2024-09-13T09:20:50.463 INFO:teuthology.orchestra.run.smithi190.stdout:(11/30): python3-rbd-18.2.4-820.g48a7de7e.el9.x 5.0 MB/s | 299 kB 00:00 2024-09-13T09:20:50.488 INFO:teuthology.orchestra.run.smithi190.stdout:(12/30): python3-rgw-18.2.4-820.g48a7de7e.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-13T09:20:50.672 INFO:teuthology.orchestra.run.smithi190.stdout:(13/30): ceph-common-18.2.4-820.g48a7de7e.el9.x 15 MB/s | 18 MB 00:01 2024-09-13T09:20:51.247 INFO:teuthology.orchestra.run.smithi190.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 79 kB/s | 45 kB 00:00 2024-09-13T09:20:51.281 INFO:teuthology.orchestra.run.smithi190.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 131 kB/s | 107 kB 00:00 2024-09-13T09:20:51.373 INFO:teuthology.orchestra.run.smithi190.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 181 kB/s | 160 kB 00:00 2024-09-13T09:20:51.990 INFO:teuthology.orchestra.run.smithi190.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 68 kB/s | 42 kB 00:00 2024-09-13T09:20:52.040 INFO:teuthology.orchestra.run.smithi190.stdout:(18/30): fmt-8.1.1-5.el9.x86_64.rpm 2.2 MB/s | 111 kB 00:00 2024-09-13T09:20:52.065 INFO:teuthology.orchestra.run.smithi190.stdout:(19/30): gperftools-libs-2.9.1-3.el9.x86_64.rpm 12 MB/s | 308 kB 00:00 2024-09-13T09:20:52.090 INFO:teuthology.orchestra.run.smithi190.stdout:(20/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 361 kB/s | 292 kB 00:00 2024-09-13T09:20:52.124 INFO:teuthology.orchestra.run.smithi190.stdout:(21/30): librdkafka-1.6.1-102.el9.x86_64.rpm 757 kB/s | 662 kB 00:00 2024-09-13T09:20:52.149 INFO:teuthology.orchestra.run.smithi190.stdout:(22/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 445 kB/s | 26 kB 00:00 2024-09-13T09:20:52.183 INFO:teuthology.orchestra.run.smithi190.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-13T09:20:52.201 INFO:teuthology.orchestra.run.smithi190.stdout:(24/30): liboath-2.6.7-2.el9.x86_64.rpm 661 kB/s | 49 kB 00:00 2024-09-13T09:20:52.234 INFO:teuthology.orchestra.run.smithi190.stdout:(25/30): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-13T09:20:52.301 INFO:teuthology.orchestra.run.smithi190.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-13T09:20:52.343 INFO:teuthology.orchestra.run.smithi190.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-13T09:20:52.401 INFO:teuthology.orchestra.run.smithi190.stdout:(28/30): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-13T09:20:52.610 INFO:teuthology.orchestra.run.smithi190.stdout:(29/30): librbd1-18.2.4-820.g48a7de7e.el9.x86_6 11 MB/s | 3.0 MB 00:00 2024-09-13T09:20:52.660 INFO:teuthology.orchestra.run.smithi190.stdout:(30/30): librados2-18.2.4-820.g48a7de7e.el9.x86 9.2 MB/s | 3.3 MB 00:00 2024-09-13T09:20:52.666 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:20:52.666 INFO:teuthology.orchestra.run.smithi190.stdout:Total 16 MB/s | 53 MB 00:03 2024-09-13T09:20:52.739 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-13T09:20:52.774 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-13T09:20:52.775 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-13T09:20:53.412 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-13T09:20:53.413 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-13T09:20:54.734 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-09-13T09:20:54.759 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-13T09:20:54.836 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-13T09:20:54.865 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-13T09:20:54.922 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-13T09:20:55.288 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-13T09:20:55.339 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librados2-2:18.2.4-820.g48a7de7e.el9.x86_64 5/32 2024-09-13T09:20:55.431 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:18.2.4-820.g48a7de7e.el9.x86_64 5/32 2024-09-13T09:20:55.524 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rados-2:18.2.4-820.g48a7de7e.el9.x86_64 6/32 2024-09-13T09:20:55.558 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs2-2:18.2.4-820.g48a7de7e.el9.x86_64 7/32 2024-09-13T09:20:55.620 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs2-2:18.2.4-820.g48a7de7e.el9.x86_64 7/32 2024-09-13T09:20:55.665 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-13T09:20:55.708 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-argparse-2:18.2.4-820.g48a7de7e.el9.x 9/32 2024-09-13T09:20:55.776 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cephfs-2:18.2.4-820.g48a7de7e.el9.x86_64 10/32 2024-09-13T09:20:55.801 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libradosstriper1-2:18.2.4-820.g48a7de7e.el9.x86_64 11/32 2024-09-13T09:20:55.850 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libradosstriper1-2:18.2.4-820.g48a7de7e.el9.x86_64 11/32 2024-09-13T09:20:55.956 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-13T09:20:55.993 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-13T09:20:56.033 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-13T09:20:56.074 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/32 2024-09-13T09:20:56.109 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-13T09:20:56.145 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-13T09:20:56.181 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-13T09:20:56.476 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-13T09:20:56.526 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librbd1-2:18.2.4-820.g48a7de7e.el9.x86_64 20/32 2024-09-13T09:20:56.622 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:18.2.4-820.g48a7de7e.el9.x86_64 20/32 2024-09-13T09:20:56.654 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rbd-2:18.2.4-820.g48a7de7e.el9.x86_64 21/32 2024-09-13T09:20:56.698 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-13T09:20:57.103 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-13T09:20:57.126 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:20:57.135 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librgw2-2:18.2.4-820.g48a7de7e.el9.x86_64 24/32 2024-09-13T09:20:57.144 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:20:57.144 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T09:20:57.144 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:20:57.144 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T09:20:57.144 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 7.7 M 2024-09-13T09:20:57.144 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrading: 2024-09-13T09:20:57.144 INFO:teuthology.orchestra.run.smithi122.stdout: librados2 x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 3.3 M 2024-09-13T09:20:57.144 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1 x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 3.0 M 2024-09-13T09:20:57.144 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T09:20:57.144 INFO:teuthology.orchestra.run.smithi122.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T09:20:57.144 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 5.1 M 2024-09-13T09:20:57.144 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 18 M 2024-09-13T09:20:57.145 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 25 k 2024-09-13T09:20:57.145 INFO:teuthology.orchestra.run.smithi122.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-13T09:20:57.145 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T09:20:57.145 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T09:20:57.145 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T09:20:57.145 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2 x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 711 k 2024-09-13T09:20:57.145 INFO:teuthology.orchestra.run.smithi122.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T09:20:57.145 INFO:teuthology.orchestra.run.smithi122.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T09:20:57.145 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T09:20:57.145 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1 x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 478 k 2024-09-13T09:20:57.145 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T09:20:57.145 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2 x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 4.5 M 2024-09-13T09:20:57.145 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T09:20:57.146 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T09:20:57.146 INFO:teuthology.orchestra.run.smithi122.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T09:20:57.146 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 45 k 2024-09-13T09:20:57.146 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 129 k 2024-09-13T09:20:57.146 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 162 k 2024-09-13T09:20:57.146 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T09:20:57.146 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 321 k 2024-09-13T09:20:57.146 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 299 k 2024-09-13T09:20:57.146 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 99 k 2024-09-13T09:20:57.146 INFO:teuthology.orchestra.run.smithi122.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T09:20:57.146 INFO:teuthology.orchestra.run.smithi122.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T09:20:57.146 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:20:57.146 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T09:20:57.146 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:20:57.147 INFO:teuthology.orchestra.run.smithi122.stdout:Install 28 Packages 2024-09-13T09:20:57.147 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrade 2 Packages 2024-09-13T09:20:57.147 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:20:57.147 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 53 M 2024-09-13T09:20:57.147 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T09:20:57.207 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librgw2-2:18.2.4-820.g48a7de7e.el9.x86_64 24/32 2024-09-13T09:20:57.283 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rgw-2:18.2.4-820.g48a7de7e.el9.x86_64 25/32 2024-09-13T09:20:57.312 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-common-2:18.2.4-820.g48a7de7e.el9.x86 26/32 2024-09-13T09:20:57.986 INFO:teuthology.orchestra.run.smithi122.stdout:(1/30): ceph-base-18.2.4-820.g48a7de7e.el9.x86_ 10 MB/s | 5.1 MB 00:00 2024-09-13T09:20:58.053 INFO:teuthology.orchestra.run.smithi122.stdout:(2/30): ceph-selinux-18.2.4-820.g48a7de7e.el9.x 377 kB/s | 25 kB 00:00 2024-09-13T09:20:58.153 INFO:teuthology.orchestra.run.smithi122.stdout:(3/30): ceph-radosgw-18.2.4-820.g48a7de7e.el9.x 11 MB/s | 7.7 MB 00:00 2024-09-13T09:20:58.187 INFO:teuthology.orchestra.run.smithi122.stdout:(4/30): libcephfs2-18.2.4-820.g48a7de7e.el9.x86 5.2 MB/s | 711 kB 00:00 2024-09-13T09:20:58.212 INFO:teuthology.orchestra.run.smithi122.stdout:(5/30): libradosstriper1-18.2.4-820.g48a7de7e.e 8.0 MB/s | 478 kB 00:00 2024-09-13T09:20:58.262 INFO:teuthology.orchestra.run.smithi122.stdout:(6/30): python3-ceph-argparse-18.2.4-820.g48a7d 897 kB/s | 45 kB 00:00 2024-09-13T09:20:58.354 INFO:teuthology.orchestra.run.smithi122.stdout:(7/30): python3-ceph-common-18.2.4-820.g48a7de7 1.4 MB/s | 129 kB 00:00 2024-09-13T09:20:58.429 INFO:teuthology.orchestra.run.smithi122.stdout:(8/30): librgw2-18.2.4-820.g48a7de7e.el9.x86_64 18 MB/s | 4.5 MB 00:00 2024-09-13T09:20:58.454 INFO:teuthology.orchestra.run.smithi122.stdout:(9/30): python3-cephfs-18.2.4-820.g48a7de7e.el9 1.6 MB/s | 162 kB 00:00 2024-09-13T09:20:58.480 INFO:teuthology.orchestra.run.smithi122.stdout:(10/30): python3-rados-18.2.4-820.g48a7de7e.el9 6.3 MB/s | 321 kB 00:00 2024-09-13T09:20:58.505 INFO:teuthology.orchestra.run.smithi122.stdout:(11/30): python3-rbd-18.2.4-820.g48a7de7e.el9.x 5.8 MB/s | 299 kB 00:00 2024-09-13T09:20:58.530 INFO:teuthology.orchestra.run.smithi122.stdout:(12/30): python3-rgw-18.2.4-820.g48a7de7e.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-13T09:20:58.714 INFO:teuthology.orchestra.run.smithi122.stdout:(13/30): ceph-common-18.2.4-820.g48a7de7e.el9.x 15 MB/s | 18 MB 00:01 2024-09-13T09:20:59.298 INFO:teuthology.orchestra.run.smithi122.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 135 kB/s | 107 kB 00:00 2024-09-13T09:20:59.331 INFO:teuthology.orchestra.run.smithi122.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 73 kB/s | 45 kB 00:00 2024-09-13T09:20:59.356 INFO:teuthology.orchestra.run.smithi122.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 194 kB/s | 160 kB 00:00 2024-09-13T09:20:59.431 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:18.2.4-820.g48a7de7e.el9.x86_64 27/32 2024-09-13T09:20:59.467 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-common-2:18.2.4-820.g48a7de7e.el9.x86_64 27/32 2024-09-13T09:20:59.932 INFO:teuthology.orchestra.run.smithi122.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 73 kB/s | 42 kB 00:00 2024-09-13T09:20:59.976 INFO:teuthology.orchestra.run.smithi122.stdout:(18/30): fmt-8.1.1-5.el9.x86_64.rpm 2.6 MB/s | 111 kB 00:00 2024-09-13T09:20:59.998 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:18.2.4-820.g48a7de7e.el9.x86_64 27/32 2024-09-13T09:20:59.999 INFO:teuthology.orchestra.run.smithi122.stdout:(19/30): gperftools-libs-2.9.1-3.el9.x86_64.rpm 12 MB/s | 308 kB 00:00 2024-09-13T09:21:00.037 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-base-2:18.2.4-820.g48a7de7e.el9.x86_64 28/32 2024-09-13T09:21:00.107 INFO:teuthology.orchestra.run.smithi122.stdout:(20/30): libarrow-9.0.0-11.el9.x86_64.rpm 41 MB/s | 4.4 MB 00:00 2024-09-13T09:21:00.133 INFO:teuthology.orchestra.run.smithi122.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-13T09:21:00.136 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-base-2:18.2.4-820.g48a7de7e.el9.x86_64 28/32 2024-09-13T09:21:00.137 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T09:21:00.137 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T09:21:00.137 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:00.158 INFO:teuthology.orchestra.run.smithi122.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-13T09:21:00.172 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-selinux-2:18.2.4-820.g48a7de7e.el9.x86_64 29/32 2024-09-13T09:21:00.183 INFO:teuthology.orchestra.run.smithi122.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-13T09:21:00.217 INFO:teuthology.orchestra.run.smithi122.stdout:(24/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 25 MB/s | 839 kB 00:00 2024-09-13T09:21:00.242 INFO:teuthology.orchestra.run.smithi122.stdout:(25/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 321 kB/s | 292 kB 00:00 2024-09-13T09:21:00.284 INFO:teuthology.orchestra.run.smithi122.stdout:(26/30): librdkafka-1.6.1-102.el9.x86_64.rpm 672 kB/s | 662 kB 00:00 2024-09-13T09:21:00.309 INFO:teuthology.orchestra.run.smithi122.stdout:(27/30): re2-20211101-20.el9.x86_64.rpm 2.0 MB/s | 191 kB 00:00 2024-09-13T09:21:00.376 INFO:teuthology.orchestra.run.smithi122.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-09-13T09:21:00.560 INFO:teuthology.orchestra.run.smithi122.stdout:(29/30): librbd1-18.2.4-820.g48a7de7e.el9.x86_6 12 MB/s | 3.0 MB 00:00 2024-09-13T09:21:00.627 INFO:teuthology.orchestra.run.smithi122.stdout:(30/30): librados2-18.2.4-820.g48a7de7e.el9.x86 9.7 MB/s | 3.3 MB 00:00 2024-09-13T09:21:00.631 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:21:00.631 INFO:teuthology.orchestra.run.smithi122.stdout:Total 15 MB/s | 53 MB 00:03 2024-09-13T09:21:00.701 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T09:21:00.736 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T09:21:00.736 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T09:21:01.351 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T09:21:01.352 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T09:21:02.749 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T09:21:02.834 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-13T09:21:02.865 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-13T09:21:02.923 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-13T09:21:03.239 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-13T09:21:03.364 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librados2-2:18.2.4-820.g48a7de7e.el9.x86_64 5/32 2024-09-13T09:21:03.456 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:18.2.4-820.g48a7de7e.el9.x86_64 5/32 2024-09-13T09:21:03.549 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rados-2:18.2.4-820.g48a7de7e.el9.x86_64 6/32 2024-09-13T09:21:03.574 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs2-2:18.2.4-820.g48a7de7e.el9.x86_64 7/32 2024-09-13T09:21:03.628 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs2-2:18.2.4-820.g48a7de7e.el9.x86_64 7/32 2024-09-13T09:21:03.657 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-13T09:21:03.692 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-argparse-2:18.2.4-820.g48a7de7e.el9.x 9/32 2024-09-13T09:21:03.758 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cephfs-2:18.2.4-820.g48a7de7e.el9.x86_64 10/32 2024-09-13T09:21:03.784 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libradosstriper1-2:18.2.4-820.g48a7de7e.el9.x86_64 11/32 2024-09-13T09:21:03.850 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libradosstriper1-2:18.2.4-820.g48a7de7e.el9.x86_64 11/32 2024-09-13T09:21:03.964 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-13T09:21:04.001 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-13T09:21:04.041 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-13T09:21:04.082 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/32 2024-09-13T09:21:04.108 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-13T09:21:04.145 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-13T09:21:04.187 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-13T09:21:04.485 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-13T09:21:04.532 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librbd1-2:18.2.4-820.g48a7de7e.el9.x86_64 20/32 2024-09-13T09:21:04.622 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:18.2.4-820.g48a7de7e.el9.x86_64 20/32 2024-09-13T09:21:04.653 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rbd-2:18.2.4-820.g48a7de7e.el9.x86_64 21/32 2024-09-13T09:21:04.689 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-13T09:21:05.093 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-13T09:21:05.117 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librgw2-2:18.2.4-820.g48a7de7e.el9.x86_64 24/32 2024-09-13T09:21:05.189 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librgw2-2:18.2.4-820.g48a7de7e.el9.x86_64 24/32 2024-09-13T09:21:05.274 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rgw-2:18.2.4-820.g48a7de7e.el9.x86_64 25/32 2024-09-13T09:21:05.311 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-common-2:18.2.4-820.g48a7de7e.el9.x86 26/32 2024-09-13T09:21:07.446 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:18.2.4-820.g48a7de7e.el9.x86_64 27/32 2024-09-13T09:21:07.483 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-common-2:18.2.4-820.g48a7de7e.el9.x86_64 27/32 2024-09-13T09:21:08.021 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:18.2.4-820.g48a7de7e.el9.x86_64 27/32 2024-09-13T09:21:08.069 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-base-2:18.2.4-820.g48a7de7e.el9.x86_64 28/32 2024-09-13T09:21:08.169 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-base-2:18.2.4-820.g48a7de7e.el9.x86_64 28/32 2024-09-13T09:21:08.170 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T09:21:08.170 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T09:21:08.170 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:08.196 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-selinux-2:18.2.4-820.g48a7de7e.el9.x86_64 29/32 2024-09-13T09:21:09.927 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-selinux-2:18.2.4-820.g48a7de7e.el9.x86_64 29/32 2024-09-13T09:21:09.927 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /sys 2024-09-13T09:21:09.927 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /proc 2024-09-13T09:21:09.927 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /mnt 2024-09-13T09:21:09.927 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /var/tmp 2024-09-13T09:21:09.928 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /home 2024-09-13T09:21:09.928 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /root 2024-09-13T09:21:09.928 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /tmp 2024-09-13T09:21:09.928 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:10.665 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-radosgw-2:18.2.4-820.g48a7de7e.el9.x86_64 30/32 2024-09-13T09:21:10.690 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-820.g48a7de7e.el9.x86_64 30/32 2024-09-13T09:21:10.691 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T09:21:10.691 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T09:21:10.691 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T09:21:10.691 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T09:21:10.691 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:10.692 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-13T09:21:10.776 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-13T09:21:10.776 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T09:21:11.990 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T09:21:11.990 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-base-2:18.2.4-820.g48a7de7e.el9.x86_64 1/32 2024-09-13T09:21:11.990 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-common-2:18.2.4-820.g48a7de7e.el9.x86_64 2/32 2024-09-13T09:21:11.990 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-radosgw-2:18.2.4-820.g48a7de7e.el9.x86_64 3/32 2024-09-13T09:21:11.990 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-selinux-2:18.2.4-820.g48a7de7e.el9.x86_64 4/32 2024-09-13T09:21:11.990 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs2-2:18.2.4-820.g48a7de7e.el9.x86_64 5/32 2024-09-13T09:21:11.990 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libradosstriper1-2:18.2.4-820.g48a7de7e.el9.x86_64 6/32 2024-09-13T09:21:11.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librgw2-2:18.2.4-820.g48a7de7e.el9.x86_64 7/32 2024-09-13T09:21:11.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-argparse-2:18.2.4-820.g48a7de7e.el9.x 8/32 2024-09-13T09:21:11.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-common-2:18.2.4-820.g48a7de7e.el9.x86 9/32 2024-09-13T09:21:11.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cephfs-2:18.2.4-820.g48a7de7e.el9.x86_64 10/32 2024-09-13T09:21:11.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rados-2:18.2.4-820.g48a7de7e.el9.x86_64 11/32 2024-09-13T09:21:11.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rbd-2:18.2.4-820.g48a7de7e.el9.x86_64 12/32 2024-09-13T09:21:11.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rgw-2:18.2.4-820.g48a7de7e.el9.x86_64 13/32 2024-09-13T09:21:11.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-13T09:21:11.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-13T09:21:11.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-13T09:21:11.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-13T09:21:11.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-13T09:21:11.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-13T09:21:11.992 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-13T09:21:11.992 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 21/32 2024-09-13T09:21:11.992 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-13T09:21:11.992 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-13T09:21:11.992 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-13T09:21:11.992 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-13T09:21:11.992 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-13T09:21:11.992 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-13T09:21:11.992 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-13T09:21:11.993 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:18.2.4-820.g48a7de7e.el9.x86_64 29/32 2024-09-13T09:21:11.993 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-13T09:21:11.993 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:18.2.4-820.g48a7de7e.el9.x86_64 31/32 2024-09-13T09:21:13.125 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T09:21:13.126 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:13.126 INFO:teuthology.orchestra.run.smithi190.stdout:Upgraded: 2024-09-13T09:21:13.126 INFO:teuthology.orchestra.run.smithi190.stdout: librados2-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:13.126 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:13.126 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-13T09:21:13.126 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T09:21:13.126 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:13.126 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:13.126 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:13.126 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:13.126 INFO:teuthology.orchestra.run.smithi190.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-13T09:21:13.126 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T09:21:13.126 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T09:21:13.127 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T09:21:13.127 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:13.127 INFO:teuthology.orchestra.run.smithi190.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T09:21:13.127 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T09:21:13.127 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T09:21:13.127 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:13.127 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T09:21:13.127 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:13.127 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T09:21:13.127 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T09:21:13.127 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T09:21:13.127 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:13.127 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:13.127 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:13.128 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T09:21:13.128 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:13.128 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:13.128 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:13.128 INFO:teuthology.orchestra.run.smithi190.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T09:21:13.128 INFO:teuthology.orchestra.run.smithi190.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T09:21:13.128 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:13.128 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:21:13.471 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-test 2024-09-13T09:21:14.037 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:28 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:21:14.170 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:21:14.171 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:21:14.171 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-13T09:21:14.171 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:21:14.171 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-13T09:21:14.171 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 40 M 2024-09-13T09:21:14.171 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-13T09:21:14.172 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 166 k 2024-09-13T09:21:14.172 INFO:teuthology.orchestra.run.smithi190.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T09:21:14.172 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T09:21:14.172 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:14.172 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-13T09:21:14.172 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:21:14.172 INFO:teuthology.orchestra.run.smithi190.stdout:Install 4 Packages 2024-09-13T09:21:14.172 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:14.173 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 40 M 2024-09-13T09:21:14.173 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 164 M 2024-09-13T09:21:14.173 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-13T09:21:14.514 INFO:teuthology.orchestra.run.smithi190.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-09-13T09:21:14.540 INFO:teuthology.orchestra.run.smithi190.stdout:(2/4): libcephsqlite-18.2.4-820.g48a7de7e.el9.x 706 kB/s | 166 kB 00:00 2024-09-13T09:21:14.565 INFO:teuthology.orchestra.run.smithi190.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-13T09:21:15.615 INFO:teuthology.orchestra.run.smithi190.stdout:(4/4): ceph-test-18.2.4-820.g48a7de7e.el9.x86_6 30 MB/s | 40 MB 00:01 2024-09-13T09:21:15.617 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:21:15.617 INFO:teuthology.orchestra.run.smithi190.stdout:Total 28 MB/s | 40 MB 00:01 2024-09-13T09:21:15.645 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-13T09:21:15.665 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-13T09:21:15.666 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-13T09:21:15.995 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-13T09:21:15.995 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-13T09:21:16.479 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-13T09:21:16.574 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T09:21:16.615 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T09:21:16.642 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephsqlite-2:18.2.4-820.g48a7de7e.el9.x86_64 3/4 2024-09-13T09:21:17.968 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-selinux-2:18.2.4-820.g48a7de7e.el9.x86_64 29/32 2024-09-13T09:21:17.968 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /sys 2024-09-13T09:21:17.968 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /proc 2024-09-13T09:21:17.969 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /mnt 2024-09-13T09:21:17.969 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /var/tmp 2024-09-13T09:21:17.969 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /home 2024-09-13T09:21:17.969 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /root 2024-09-13T09:21:17.969 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /tmp 2024-09-13T09:21:17.969 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:18.732 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-radosgw-2:18.2.4-820.g48a7de7e.el9.x86_64 30/32 2024-09-13T09:21:18.757 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-820.g48a7de7e.el9.x86_64 30/32 2024-09-13T09:21:18.757 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T09:21:18.757 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T09:21:18.757 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T09:21:18.757 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T09:21:18.757 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:18.759 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-13T09:21:18.834 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-13T09:21:18.834 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T09:21:19.998 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T09:21:19.998 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-base-2:18.2.4-820.g48a7de7e.el9.x86_64 1/32 2024-09-13T09:21:19.998 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-common-2:18.2.4-820.g48a7de7e.el9.x86_64 2/32 2024-09-13T09:21:19.998 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-radosgw-2:18.2.4-820.g48a7de7e.el9.x86_64 3/32 2024-09-13T09:21:19.998 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-selinux-2:18.2.4-820.g48a7de7e.el9.x86_64 4/32 2024-09-13T09:21:19.998 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs2-2:18.2.4-820.g48a7de7e.el9.x86_64 5/32 2024-09-13T09:21:19.998 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libradosstriper1-2:18.2.4-820.g48a7de7e.el9.x86_64 6/32 2024-09-13T09:21:19.998 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librgw2-2:18.2.4-820.g48a7de7e.el9.x86_64 7/32 2024-09-13T09:21:19.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-argparse-2:18.2.4-820.g48a7de7e.el9.x 8/32 2024-09-13T09:21:19.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-common-2:18.2.4-820.g48a7de7e.el9.x86 9/32 2024-09-13T09:21:19.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cephfs-2:18.2.4-820.g48a7de7e.el9.x86_64 10/32 2024-09-13T09:21:19.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rados-2:18.2.4-820.g48a7de7e.el9.x86_64 11/32 2024-09-13T09:21:19.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rbd-2:18.2.4-820.g48a7de7e.el9.x86_64 12/32 2024-09-13T09:21:19.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rgw-2:18.2.4-820.g48a7de7e.el9.x86_64 13/32 2024-09-13T09:21:19.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-13T09:21:19.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-13T09:21:19.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-13T09:21:19.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-13T09:21:19.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-13T09:21:19.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-13T09:21:19.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-13T09:21:19.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 21/32 2024-09-13T09:21:19.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-13T09:21:20.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-13T09:21:20.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-13T09:21:20.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-13T09:21:20.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-13T09:21:20.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-13T09:21:20.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-13T09:21:20.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:18.2.4-820.g48a7de7e.el9.x86_64 29/32 2024-09-13T09:21:20.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-13T09:21:20.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:18.2.4-820.g48a7de7e.el9.x86_64 31/32 2024-09-13T09:21:20.340 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephsqlite-2:18.2.4-820.g48a7de7e.el9.x86_64 3/4 2024-09-13T09:21:20.415 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-test-2:18.2.4-820.g48a7de7e.el9.x86_64 4/4 2024-09-13T09:21:21.007 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T09:21:21.007 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:21.007 INFO:teuthology.orchestra.run.smithi122.stdout:Upgraded: 2024-09-13T09:21:21.007 INFO:teuthology.orchestra.run.smithi122.stdout: librados2-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:21.007 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:21.008 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T09:21:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T09:21:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-13T09:21:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T09:21:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T09:21:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T09:21:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T09:21:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T09:21:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T09:21:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T09:21:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T09:21:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T09:21:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T09:21:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T09:21:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T09:21:21.010 INFO:teuthology.orchestra.run.smithi122.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T09:21:21.010 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:21.010 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:21:21.393 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-test 2024-09-13T09:21:21.957 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:27 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:21:22.093 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:21:22.093 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:21:22.093 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T09:21:22.093 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:21:22.093 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T09:21:22.093 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 40 M 2024-09-13T09:21:22.094 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T09:21:22.094 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 166 k 2024-09-13T09:21:22.094 INFO:teuthology.orchestra.run.smithi122.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T09:21:22.094 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T09:21:22.094 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:22.094 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T09:21:22.094 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:21:22.094 INFO:teuthology.orchestra.run.smithi122.stdout:Install 4 Packages 2024-09-13T09:21:22.094 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:22.094 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 40 M 2024-09-13T09:21:22.095 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 164 M 2024-09-13T09:21:22.095 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T09:21:22.314 INFO:teuthology.orchestra.run.smithi122.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-13T09:21:22.339 INFO:teuthology.orchestra.run.smithi122.stdout:(2/4): libcephsqlite-18.2.4-820.g48a7de7e.el9.x 742 kB/s | 166 kB 00:00 2024-09-13T09:21:22.356 INFO:teuthology.orchestra.run.smithi122.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-13T09:21:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-test-2:18.2.4-820.g48a7de7e.el9.x86_64 4/4 2024-09-13T09:21:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-test-2:18.2.4-820.g48a7de7e.el9.x86_64 1/4 2024-09-13T09:21:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephsqlite-2:18.2.4-820.g48a7de7e.el9.x86_64 2/4 2024-09-13T09:21:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T09:21:22.683 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T09:21:22.684 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:22.684 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-13T09:21:22.684 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:22.684 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:22.684 INFO:teuthology.orchestra.run.smithi190.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T09:21:22.684 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T09:21:22.684 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:22.684 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:21:22.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph 2024-09-13T09:21:23.423 INFO:teuthology.orchestra.run.smithi122.stdout:(4/4): ceph-test-18.2.4-820.g48a7de7e.el9.x86_6 30 MB/s | 40 MB 00:01 2024-09-13T09:21:23.425 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:21:23.426 INFO:teuthology.orchestra.run.smithi122.stdout:Total 30 MB/s | 40 MB 00:01 2024-09-13T09:21:23.453 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T09:21:23.472 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T09:21:23.472 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T09:21:23.543 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:37 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:21:23.680 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:21:23.682 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:21:23.683 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-13T09:21:23.683 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:21:23.683 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-13T09:21:23.683 INFO:teuthology.orchestra.run.smithi190.stdout: ceph x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 6.5 k 2024-09-13T09:21:23.683 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-13T09:21:23.683 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 2.1 M 2024-09-13T09:21:23.683 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 1.5 M 2024-09-13T09:21:23.683 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core noarch 2:18.2.4-820.g48a7de7e.el9 ceph-noarch 246 k 2024-09-13T09:21:23.683 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 4.7 M 2024-09-13T09:21:23.683 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 17 M 2024-09-13T09:21:23.683 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T09:21:23.684 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T09:21:23.684 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T09:21:23.684 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T09:21:23.684 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T09:21:23.684 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T09:21:23.684 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T09:21:23.684 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-13T09:21:23.684 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T09:21:23.684 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T09:21:23.684 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T09:21:23.684 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T09:21:23.684 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T09:21:23.685 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T09:21:23.685 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T09:21:23.685 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T09:21:23.685 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T09:21:23.685 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T09:21:23.685 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T09:21:23.685 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T09:21:23.685 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T09:21:23.685 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T09:21:23.685 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T09:21:23.685 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T09:21:23.685 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T09:21:23.685 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T09:21:23.685 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T09:21:23.686 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:23.686 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-13T09:21:23.686 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:21:23.686 INFO:teuthology.orchestra.run.smithi190.stdout:Install 33 Packages 2024-09-13T09:21:23.686 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:23.686 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 29 M 2024-09-13T09:21:23.686 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 104 M 2024-09-13T09:21:23.687 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-13T09:21:23.804 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T09:21:23.804 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T09:21:24.141 INFO:teuthology.orchestra.run.smithi190.stdout:(1/33): ceph-18.2.4-820.g48a7de7e.el9.x86_64.rp 51 kB/s | 6.5 kB 00:00 2024-09-13T09:21:24.286 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T09:21:24.333 INFO:teuthology.orchestra.run.smithi190.stdout:(2/33): ceph-mgr-18.2.4-820.g48a7de7e.el9.x86_6 4.6 MB/s | 1.5 MB 00:00 2024-09-13T09:21:24.358 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T09:21:24.375 INFO:teuthology.orchestra.run.smithi190.stdout:(3/33): ceph-mds-18.2.4-820.g48a7de7e.el9.x86_6 5.8 MB/s | 2.1 MB 00:00 2024-09-13T09:21:24.406 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T09:21:24.433 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephsqlite-2:18.2.4-820.g48a7de7e.el9.x86_64 3/4 2024-09-13T09:21:24.476 INFO:teuthology.orchestra.run.smithi190.stdout:(4/33): ceph-mon-18.2.4-820.g48a7de7e.el9.x86_6 14 MB/s | 4.7 MB 00:00 2024-09-13T09:21:24.501 INFO:teuthology.orchestra.run.smithi190.stdout:(5/33): ceph-mgr-modules-core-18.2.4-820.g48a7d 1.9 MB/s | 246 kB 00:00 2024-09-13T09:21:24.710 INFO:teuthology.orchestra.run.smithi190.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-13T09:21:24.893 INFO:teuthology.orchestra.run.smithi190.stdout:(7/33): ceph-osd-18.2.4-820.g48a7de7e.el9.x86_6 30 MB/s | 17 MB 00:00 2024-09-13T09:21:24.918 INFO:teuthology.orchestra.run.smithi190.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 511 kB/s | 106 kB 00:00 2024-09-13T09:21:24.952 INFO:teuthology.orchestra.run.smithi190.stdout:(9/33): python3-cryptography-36.0.1-4.el9.x86_6 2.7 MB/s | 1.2 MB 00:00 2024-09-13T09:21:24.985 INFO:teuthology.orchestra.run.smithi190.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-09-13T09:21:25.019 INFO:teuthology.orchestra.run.smithi190.stdout:(11/33): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-13T09:21:25.128 INFO:teuthology.orchestra.run.smithi190.stdout:(12/33): python3-pycparser-2.20-6.el9.noarch.rp 577 kB/s | 135 kB 00:00 2024-09-13T09:21:25.328 INFO:teuthology.orchestra.run.smithi190.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 112 kB/s | 35 kB 00:00 2024-09-13T09:21:25.503 INFO:teuthology.orchestra.run.smithi190.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 136 kB/s | 51 kB 00:00 2024-09-13T09:21:25.529 INFO:teuthology.orchestra.run.smithi190.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 208 kB/s | 42 kB 00:00 2024-09-13T09:21:25.571 INFO:teuthology.orchestra.run.smithi190.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 648 kB/s | 43 kB 00:00 2024-09-13T09:21:25.596 INFO:teuthology.orchestra.run.smithi190.stdout:(17/33): python3-mako-1.1.4-6.el9.noarch.rpm 282 kB/s | 172 kB 00:00 2024-09-13T09:21:25.630 INFO:teuthology.orchestra.run.smithi190.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-09-13T09:21:25.647 INFO:teuthology.orchestra.run.smithi190.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-09-13T09:21:25.663 INFO:teuthology.orchestra.run.smithi190.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 560 kB/s | 18 kB 00:00 2024-09-13T09:21:25.689 INFO:teuthology.orchestra.run.smithi190.stdout:(21/33): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-09-13T09:21:25.706 INFO:teuthology.orchestra.run.smithi190.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 466 kB/s | 19 kB 00:00 2024-09-13T09:21:25.723 INFO:teuthology.orchestra.run.smithi190.stdout:(23/33): python3-jaraco-collections-3.0.0-8.el9 394 kB/s | 23 kB 00:00 2024-09-13T09:21:25.740 INFO:teuthology.orchestra.run.smithi190.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 389 kB/s | 20 kB 00:00 2024-09-13T09:21:25.765 INFO:teuthology.orchestra.run.smithi190.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 786 kB/s | 46 kB 00:00 2024-09-13T09:21:25.790 INFO:teuthology.orchestra.run.smithi190.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-09-13T09:21:25.815 INFO:teuthology.orchestra.run.smithi190.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-09-13T09:21:25.841 INFO:teuthology.orchestra.run.smithi190.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-13T09:21:25.866 INFO:teuthology.orchestra.run.smithi190.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-13T09:21:25.891 INFO:teuthology.orchestra.run.smithi190.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-13T09:21:25.917 INFO:teuthology.orchestra.run.smithi190.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 799 kB/s | 20 kB 00:00 2024-09-13T09:21:25.942 INFO:teuthology.orchestra.run.smithi190.stdout:(32/33): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-13T09:21:25.967 INFO:teuthology.orchestra.run.smithi190.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-13T09:21:25.972 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:21:25.972 INFO:teuthology.orchestra.run.smithi190.stdout:Total 13 MB/s | 29 MB 00:02 2024-09-13T09:21:26.094 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-13T09:21:26.131 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-13T09:21:26.131 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-13T09:21:26.529 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-13T09:21:26.530 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-13T09:21:27.763 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-13T09:21:27.836 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-13T09:21:27.869 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-13T09:21:27.910 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-13T09:21:28.002 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-13T09:21:28.074 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-13T09:21:28.125 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-13T09:21:28.133 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephsqlite-2:18.2.4-820.g48a7de7e.el9.x86_64 3/4 2024-09-13T09:21:28.168 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-13T09:21:28.206 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-13T09:21:28.207 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-test-2:18.2.4-820.g48a7de7e.el9.x86_64 4/4 2024-09-13T09:21:28.241 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-13T09:21:28.274 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-13T09:21:28.314 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-13T09:21:28.349 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-13T09:21:28.384 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-13T09:21:28.425 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-13T09:21:28.504 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-13T09:21:28.652 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-13T09:21:28.739 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-13T09:21:28.817 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-13T09:21:28.855 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-13T09:21:28.906 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-13T09:21:28.968 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-13T09:21:29.180 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-13T09:21:29.236 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-13T09:21:29.316 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-13T09:21:29.413 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-13T09:21:29.461 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-13T09:21:29.580 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-13T09:21:29.746 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-820.g48a7de7e.el9.n 28/33 2024-09-13T09:21:29.774 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-2:18.2.4-820.g48a7de7e.el9.x86_64 29/33 2024-09-13T09:21:29.807 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-2:18.2.4-820.g48a7de7e.el9.x86_64 29/33 2024-09-13T09:21:29.807 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T09:21:29.807 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T09:21:29.807 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T09:21:29.807 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T09:21:29.807 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:30.275 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-test-2:18.2.4-820.g48a7de7e.el9.x86_64 4/4 2024-09-13T09:21:30.275 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-test-2:18.2.4-820.g48a7de7e.el9.x86_64 1/4 2024-09-13T09:21:30.275 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephsqlite-2:18.2.4-820.g48a7de7e.el9.x86_64 2/4 2024-09-13T09:21:30.276 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T09:21:30.571 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T09:21:30.571 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:30.572 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T09:21:30.572 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:30.572 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:30.572 INFO:teuthology.orchestra.run.smithi122.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T09:21:30.572 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T09:21:30.572 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:30.572 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:21:30.876 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph 2024-09-13T09:21:31.235 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-osd-2:18.2.4-820.g48a7de7e.el9.x86_64 30/33 2024-09-13T09:21:31.262 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-osd-2:18.2.4-820.g48a7de7e.el9.x86_64 30/33 2024-09-13T09:21:31.262 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T09:21:31.262 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T09:21:31.262 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T09:21:31.263 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T09:21:31.263 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:31.444 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:37 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:21:31.586 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:21:31.589 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:21:31.589 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T09:21:31.589 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:21:31.589 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T09:21:31.589 INFO:teuthology.orchestra.run.smithi122.stdout: ceph x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 6.5 k 2024-09-13T09:21:31.589 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T09:21:31.589 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 2.1 M 2024-09-13T09:21:31.589 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 1.5 M 2024-09-13T09:21:31.589 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core noarch 2:18.2.4-820.g48a7de7e.el9 ceph-noarch 246 k 2024-09-13T09:21:31.590 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 4.7 M 2024-09-13T09:21:31.590 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 17 M 2024-09-13T09:21:31.590 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T09:21:31.590 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T09:21:31.590 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T09:21:31.590 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T09:21:31.590 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T09:21:31.590 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T09:21:31.590 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T09:21:31.590 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-13T09:21:31.590 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T09:21:31.590 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T09:21:31.590 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T09:21:31.590 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T09:21:31.591 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T09:21:31.591 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T09:21:31.591 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T09:21:31.591 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T09:21:31.591 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T09:21:31.591 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T09:21:31.591 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T09:21:31.591 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T09:21:31.591 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T09:21:31.591 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T09:21:31.591 INFO:teuthology.orchestra.run.smithi122.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T09:21:31.591 INFO:teuthology.orchestra.run.smithi122.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T09:21:31.591 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T09:21:31.592 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T09:21:31.592 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T09:21:31.592 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:31.592 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T09:21:31.592 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:21:31.592 INFO:teuthology.orchestra.run.smithi122.stdout:Install 33 Packages 2024-09-13T09:21:31.592 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:31.592 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 29 M 2024-09-13T09:21:31.593 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 104 M 2024-09-13T09:21:31.593 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T09:21:31.703 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mon-2:18.2.4-820.g48a7de7e.el9.x86_64 31/33 2024-09-13T09:21:31.727 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mon-2:18.2.4-820.g48a7de7e.el9.x86_64 31/33 2024-09-13T09:21:31.727 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T09:21:31.727 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T09:21:31.727 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T09:21:31.727 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T09:21:31.727 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:31.928 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mds-2:18.2.4-820.g48a7de7e.el9.x86_64 32/33 2024-09-13T09:21:31.952 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mds-2:18.2.4-820.g48a7de7e.el9.x86_64 32/33 2024-09-13T09:21:31.952 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T09:21:31.952 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T09:21:31.952 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T09:21:31.952 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T09:21:31.952 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:32.016 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-2:18.2.4-820.g48a7de7e.el9.x86_64 33/33 2024-09-13T09:21:32.024 INFO:teuthology.orchestra.run.smithi122.stdout:(1/33): ceph-18.2.4-820.g48a7de7e.el9.x86_64.rp 49 kB/s | 6.5 kB 00:00 2024-09-13T09:21:32.166 INFO:teuthology.orchestra.run.smithi122.stdout:(2/33): ceph-mgr-18.2.4-820.g48a7de7e.el9.x86_6 5.3 MB/s | 1.5 MB 00:00 2024-09-13T09:21:32.233 INFO:teuthology.orchestra.run.smithi122.stdout:(3/33): ceph-mds-18.2.4-820.g48a7de7e.el9.x86_6 6.1 MB/s | 2.1 MB 00:00 2024-09-13T09:21:32.325 INFO:teuthology.orchestra.run.smithi122.stdout:(4/33): ceph-mgr-modules-core-18.2.4-820.g48a7d 2.6 MB/s | 246 kB 00:00 2024-09-13T09:21:32.400 INFO:teuthology.orchestra.run.smithi122.stdout:(5/33): ceph-mon-18.2.4-820.g48a7de7e.el9.x86_6 12 MB/s | 4.7 MB 00:00 2024-09-13T09:21:32.484 INFO:teuthology.orchestra.run.smithi122.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 253 kB 00:00 2024-09-13T09:21:32.567 INFO:teuthology.orchestra.run.smithi122.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 106 kB 00:00 2024-09-13T09:21:32.609 INFO:teuthology.orchestra.run.smithi122.stdout:(8/33): python3-cryptography-36.0.1-4.el9.x86_6 5.9 MB/s | 1.2 MB 00:00 2024-09-13T09:21:32.801 INFO:teuthology.orchestra.run.smithi122.stdout:(9/33): ceph-osd-18.2.4-820.g48a7de7e.el9.x86_6 26 MB/s | 17 MB 00:00 2024-09-13T09:21:32.843 INFO:teuthology.orchestra.run.smithi122.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 490 kB/s | 135 kB 00:00 2024-09-13T09:21:32.877 INFO:teuthology.orchestra.run.smithi122.stdout:(11/33): python3-requests-2.25.1-8.el9.noarch.r 467 kB/s | 125 kB 00:00 2024-09-13T09:21:32.927 INFO:teuthology.orchestra.run.smithi122.stdout:(12/33): python3-urllib3-1.26.5-6.el9.noarch.rp 1.7 MB/s | 215 kB 00:00 2024-09-13T09:21:33.011 INFO:teuthology.orchestra.run.smithi122.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 260 kB/s | 35 kB 00:00 2024-09-13T09:21:33.036 INFO:teuthology.orchestra.run.smithi122.stdout:(14/33): python3-mako-1.1.4-6.el9.noarch.rpm 894 kB/s | 172 kB 00:00 2024-09-13T09:21:33.061 INFO:teuthology.orchestra.run.smithi122.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 381 kB/s | 51 kB 00:00 2024-09-13T09:21:33.086 INFO:teuthology.orchestra.run.smithi122.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-09-13T09:21:33.137 INFO:teuthology.orchestra.run.smithi122.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-13T09:21:33.162 INFO:teuthology.orchestra.run.smithi122.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 460 kB/s | 11 kB 00:00 2024-09-13T09:21:33.187 INFO:teuthology.orchestra.run.smithi122.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-13T09:21:33.213 INFO:teuthology.orchestra.run.smithi122.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-09-13T09:21:33.238 INFO:teuthology.orchestra.run.smithi122.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-13T09:21:33.263 INFO:teuthology.orchestra.run.smithi122.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-13T09:21:33.289 INFO:teuthology.orchestra.run.smithi122.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-09-13T09:21:33.314 INFO:teuthology.orchestra.run.smithi122.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 925 kB/s | 46 kB 00:00 2024-09-13T09:21:33.330 INFO:teuthology.orchestra.run.smithi122.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-13T09:21:33.356 INFO:teuthology.orchestra.run.smithi122.stdout:(26/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-09-13T09:21:33.381 INFO:teuthology.orchestra.run.smithi122.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-13T09:21:33.406 INFO:teuthology.orchestra.run.smithi122.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-13T09:21:33.432 INFO:teuthology.orchestra.run.smithi122.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 715 kB/s | 36 kB 00:00 2024-09-13T09:21:33.457 INFO:teuthology.orchestra.run.smithi122.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-09-13T09:21:33.482 INFO:teuthology.orchestra.run.smithi122.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 802 kB/s | 20 kB 00:00 2024-09-13T09:21:33.507 INFO:teuthology.orchestra.run.smithi122.stdout:(32/33): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-13T09:21:33.533 INFO:teuthology.orchestra.run.smithi122.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-13T09:21:33.537 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:21:33.537 INFO:teuthology.orchestra.run.smithi122.stdout:Total 15 MB/s | 29 MB 00:01 2024-09-13T09:21:33.658 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T09:21:33.697 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T09:21:33.697 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T09:21:33.786 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-2:18.2.4-820.g48a7de7e.el9.x86_64 33/33 2024-09-13T09:21:33.786 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-2:18.2.4-820.g48a7de7e.el9.x86_64 1/33 2024-09-13T09:21:33.786 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mds-2:18.2.4-820.g48a7de7e.el9.x86_64 2/33 2024-09-13T09:21:33.786 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-2:18.2.4-820.g48a7de7e.el9.x86_64 3/33 2024-09-13T09:21:33.786 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mon-2:18.2.4-820.g48a7de7e.el9.x86_64 4/33 2024-09-13T09:21:33.786 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-osd-2:18.2.4-820.g48a7de7e.el9.x86_64 5/33 2024-09-13T09:21:33.786 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-820.g48a7de7e.el9.n 6/33 2024-09-13T09:21:33.786 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-13T09:21:33.786 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-13T09:21:33.786 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-13T09:21:33.787 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-13T09:21:33.787 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-13T09:21:33.787 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-13T09:21:33.787 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-13T09:21:33.787 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-13T09:21:33.787 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-13T09:21:33.787 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-13T09:21:33.787 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-13T09:21:33.787 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-13T09:21:33.787 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-13T09:21:33.787 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-13T09:21:33.788 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-13T09:21:33.788 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-13T09:21:33.788 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-13T09:21:33.788 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-13T09:21:33.788 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-13T09:21:33.789 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-13T09:21:33.789 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-13T09:21:33.789 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-13T09:21:33.789 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-13T09:21:33.789 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-13T09:21:33.789 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-13T09:21:33.789 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-13T09:21:34.118 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T09:21:34.118 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T09:21:34.646 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-13T09:21:34.646 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:34.646 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-13T09:21:34.646 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:34.646 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:34.646 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:34.646 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core-2:18.2.4-820.g48a7de7e.el9.noarch 2024-09-13T09:21:34.646 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:34.646 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:34.646 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T09:21:34.647 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T09:21:34.647 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T09:21:34.647 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T09:21:34.647 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T09:21:34.647 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T09:21:34.647 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T09:21:34.647 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T09:21:34.647 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T09:21:34.647 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T09:21:34.647 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T09:21:34.647 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T09:21:34.647 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T09:21:34.647 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T09:21:34.648 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T09:21:34.648 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T09:21:34.648 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T09:21:34.648 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T09:21:34.648 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T09:21:34.648 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T09:21:34.648 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T09:21:34.648 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T09:21:34.648 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T09:21:34.648 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T09:21:34.648 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T09:21:34.648 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T09:21:34.648 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T09:21:34.649 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:34.649 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:21:35.047 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-base 2024-09-13T09:21:35.333 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T09:21:35.402 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-13T09:21:35.434 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-13T09:21:35.467 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-13T09:21:35.551 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-13T09:21:35.617 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:49 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:21:35.645 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-13T09:21:35.687 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-13T09:21:35.706 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-base-2:18.2.4-820.g48a7de7e.el9.x86_64 is already installed. 2024-09-13T09:21:35.722 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-13T09:21:35.744 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-13T09:21:35.748 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:21:35.749 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-13T09:21:35.749 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:21:35.773 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-13T09:21:35.798 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-13T09:21:35.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install cephadm 2024-09-13T09:21:35.830 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-13T09:21:35.857 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-13T09:21:35.883 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-13T09:21:35.916 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-13T09:21:36.002 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-13T09:21:36.159 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-13T09:21:36.237 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-13T09:21:36.290 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-13T09:21:36.329 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-13T09:21:36.369 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:50 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:21:36.378 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-13T09:21:36.434 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-13T09:21:36.499 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:21:36.500 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:21:36.500 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-13T09:21:36.500 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:21:36.500 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-13T09:21:36.500 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm noarch 2:18.2.4-820.g48a7de7e.el9 ceph-noarch 226 k 2024-09-13T09:21:36.500 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:36.500 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-13T09:21:36.500 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:21:36.501 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-13T09:21:36.501 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:36.501 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 226 k 2024-09-13T09:21:36.501 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 217 k 2024-09-13T09:21:36.501 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-13T09:21:36.621 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-13T09:21:36.675 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-13T09:21:36.735 INFO:teuthology.orchestra.run.smithi190.stdout:cephadm-18.2.4-820.g48a7de7e.el9.noarch.rpm 964 kB/s | 226 kB 00:00 2024-09-13T09:21:36.736 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:21:36.736 INFO:teuthology.orchestra.run.smithi190.stdout:Total 958 kB/s | 226 kB 00:00 2024-09-13T09:21:36.736 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-13T09:21:36.741 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-13T09:21:36.741 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-13T09:21:36.744 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-13T09:21:36.744 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-13T09:21:36.745 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-13T09:21:36.817 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-13T09:21:36.832 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-13T09:21:36.875 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-13T09:21:36.989 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-13T09:21:37.149 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-820.g48a7de7e.el9.n 28/33 2024-09-13T09:21:37.176 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-2:18.2.4-820.g48a7de7e.el9.x86_64 29/33 2024-09-13T09:21:37.201 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-2:18.2.4-820.g48a7de7e.el9.x86_64 29/33 2024-09-13T09:21:37.201 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T09:21:37.201 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T09:21:37.201 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T09:21:37.201 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T09:21:37.201 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:37.208 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:18.2.4-820.g48a7de7e.el9.noarch 1/1 2024-09-13T09:21:37.300 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephadm-2:18.2.4-820.g48a7de7e.el9.noarch 1/1 2024-09-13T09:21:37.678 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:18.2.4-820.g48a7de7e.el9.noarch 1/1 2024-09-13T09:21:37.929 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephadm-2:18.2.4-820.g48a7de7e.el9.noarch 1/1 2024-09-13T09:21:37.930 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:37.930 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-13T09:21:37.930 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm-2:18.2.4-820.g48a7de7e.el9.noarch 2024-09-13T09:21:37.930 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:37.930 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:21:38.201 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T09:21:38.647 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-osd-2:18.2.4-820.g48a7de7e.el9.x86_64 30/33 2024-09-13T09:21:38.687 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-osd-2:18.2.4-820.g48a7de7e.el9.x86_64 30/33 2024-09-13T09:21:38.687 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T09:21:38.687 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T09:21:38.687 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T09:21:38.687 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T09:21:38.687 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:38.773 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:52 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:21:38.905 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:21:38.906 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:21:38.906 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-09-13T09:21:38.906 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:21:38.906 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-13T09:21:38.906 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 143 k 2024-09-13T09:21:38.906 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:38.906 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-13T09:21:38.906 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:21:38.906 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-13T09:21:38.907 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:38.907 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 143 k 2024-09-13T09:21:38.907 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 395 k 2024-09-13T09:21:38.907 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-13T09:21:39.111 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-immutable-object-cache-18.2.4-820.g48a7de7 700 kB/s | 143 kB 00:00 2024-09-13T09:21:39.111 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:21:39.112 INFO:teuthology.orchestra.run.smithi190.stdout:Total 695 kB/s | 143 kB 00:00 2024-09-13T09:21:39.112 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-13T09:21:39.119 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-13T09:21:39.119 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-13T09:21:39.147 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mon-2:18.2.4-820.g48a7de7e.el9.x86_64 31/33 2024-09-13T09:21:39.172 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mon-2:18.2.4-820.g48a7de7e.el9.x86_64 31/33 2024-09-13T09:21:39.172 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T09:21:39.172 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T09:21:39.172 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T09:21:39.172 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T09:21:39.172 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:39.175 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-13T09:21:39.176 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-13T09:21:39.314 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-13T09:21:39.381 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mds-2:18.2.4-820.g48a7de7e.el9.x86_64 32/33 2024-09-13T09:21:39.390 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-820.g48a7de7e.e 1/1 2024-09-13T09:21:39.407 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mds-2:18.2.4-820.g48a7de7e.el9.x86_64 32/33 2024-09-13T09:21:39.407 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T09:21:39.407 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T09:21:39.407 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T09:21:39.407 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T09:21:39.407 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:39.416 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-820.g48a7de7e.e 1/1 2024-09-13T09:21:39.416 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T09:21:39.416 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T09:21:39.416 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:39.470 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-2:18.2.4-820.g48a7de7e.el9.x86_64 33/33 2024-09-13T09:21:40.357 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-820.g48a7de7e.e 1/1 2024-09-13T09:21:40.357 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:40.357 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-13T09:21:40.357 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:40.357 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:40.357 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:21:40.565 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr 2024-09-13T09:21:41.144 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:55 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:21:41.240 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-mgr-2:18.2.4-820.g48a7de7e.el9.x86_64 is already installed. 2024-09-13T09:21:41.282 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:21:41.283 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-13T09:21:41.283 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:21:41.305 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-2:18.2.4-820.g48a7de7e.el9.x86_64 33/33 2024-09-13T09:21:41.305 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-2:18.2.4-820.g48a7de7e.el9.x86_64 1/33 2024-09-13T09:21:41.305 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mds-2:18.2.4-820.g48a7de7e.el9.x86_64 2/33 2024-09-13T09:21:41.306 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-2:18.2.4-820.g48a7de7e.el9.x86_64 3/33 2024-09-13T09:21:41.306 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mon-2:18.2.4-820.g48a7de7e.el9.x86_64 4/33 2024-09-13T09:21:41.306 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-osd-2:18.2.4-820.g48a7de7e.el9.x86_64 5/33 2024-09-13T09:21:41.306 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-820.g48a7de7e.el9.n 6/33 2024-09-13T09:21:41.306 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-13T09:21:41.306 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-13T09:21:41.306 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-13T09:21:41.306 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-13T09:21:41.306 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-13T09:21:41.306 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-13T09:21:41.306 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-13T09:21:41.306 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-13T09:21:41.306 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-13T09:21:41.307 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-13T09:21:41.307 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-13T09:21:41.307 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-13T09:21:41.307 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-13T09:21:41.307 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-13T09:21:41.307 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-13T09:21:41.307 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-13T09:21:41.307 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-13T09:21:41.307 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-13T09:21:41.307 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-13T09:21:41.308 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-13T09:21:41.308 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-13T09:21:41.309 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-13T09:21:41.309 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-13T09:21:41.309 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-13T09:21:41.309 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-13T09:21:41.309 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-13T09:21:41.340 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T09:21:41.903 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:55 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:21:42.035 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:21:42.036 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:21:42.037 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-13T09:21:42.037 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:21:42.037 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-13T09:21:42.037 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard noarch 2:18.2.4-820.g48a7de7e.el9 ceph-noarch 3.5 M 2024-09-13T09:21:42.037 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-13T09:21:42.037 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards noarch 2:18.2.4-820.g48a7de7e.el9 ceph-noarch 24 k 2024-09-13T09:21:42.038 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts noarch 2:18.2.4-820.g48a7de7e.el9 ceph-noarch 16 k 2024-09-13T09:21:42.038 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T09:21:42.038 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T09:21:42.038 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:42.038 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-13T09:21:42.039 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:21:42.039 INFO:teuthology.orchestra.run.smithi190.stdout:Install 5 Packages 2024-09-13T09:21:42.039 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:42.039 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 3.8 M 2024-09-13T09:21:42.039 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 87 M 2024-09-13T09:21:42.039 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-13T09:21:42.248 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-13T09:21:42.248 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:42.248 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T09:21:42.248 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:42.248 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:42.248 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:42.249 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core-2:18.2.4-820.g48a7de7e.el9.noarch 2024-09-13T09:21:42.249 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:42.249 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:42.249 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T09:21:42.249 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T09:21:42.249 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T09:21:42.249 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T09:21:42.249 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T09:21:42.249 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T09:21:42.249 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T09:21:42.249 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T09:21:42.249 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T09:21:42.249 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T09:21:42.250 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T09:21:42.250 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T09:21:42.250 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T09:21:42.250 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T09:21:42.250 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T09:21:42.250 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T09:21:42.250 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T09:21:42.250 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T09:21:42.250 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T09:21:42.250 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T09:21:42.250 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T09:21:42.250 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T09:21:42.250 INFO:teuthology.orchestra.run.smithi122.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T09:21:42.250 INFO:teuthology.orchestra.run.smithi122.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T09:21:42.251 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T09:21:42.251 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T09:21:42.251 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T09:21:42.251 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:42.251 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:21:42.556 INFO:teuthology.orchestra.run.smithi190.stdout:(1/5): ceph-grafana-dashboards-18.2.4-820.g48a7 161 kB/s | 24 kB 00:00 2024-09-13T09:21:42.602 INFO:teuthology.orchestra.run.smithi190.stdout:(2/5): ceph-prometheus-alerts-18.2.4-820.g48a7d 83 kB/s | 16 kB 00:00 2024-09-13T09:21:42.635 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-base 2024-09-13T09:21:42.639 INFO:teuthology.orchestra.run.smithi190.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 368 kB/s | 31 kB 00:00 2024-09-13T09:21:42.665 INFO:teuthology.orchestra.run.smithi190.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.9 MB/s | 188 kB 00:00 2024-09-13T09:21:42.823 INFO:teuthology.orchestra.run.smithi190.stdout:(5/5): ceph-mgr-dashboard-18.2.4-820.g48a7de7e. 8.4 MB/s | 3.5 MB 00:00 2024-09-13T09:21:42.827 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:21:42.827 INFO:teuthology.orchestra.run.smithi190.stdout:Total 4.8 MB/s | 3.8 MB 00:00 2024-09-13T09:21:42.828 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-13T09:21:42.835 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-13T09:21:42.835 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-13T09:21:42.918 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-13T09:21:42.919 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-13T09:21:43.166 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-13T09:21:43.204 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:49 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:21:43.246 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-09-13T09:21:43.278 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-09-13T09:21:43.296 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-base-2:18.2.4-820.g48a7de7e.el9.x86_64 is already installed. 2024-09-13T09:21:43.316 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-820.g48a7de7e.el9.no 3/5 2024-09-13T09:21:43.338 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:21:43.339 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T09:21:43.339 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:21:43.409 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install cephadm 2024-09-13T09:21:43.972 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:49 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:21:44.103 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:21:44.104 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:21:44.104 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T09:21:44.104 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:21:44.104 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T09:21:44.104 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm noarch 2:18.2.4-820.g48a7de7e.el9 ceph-noarch 226 k 2024-09-13T09:21:44.104 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:44.104 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T09:21:44.104 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:21:44.104 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T09:21:44.104 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:44.105 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 226 k 2024-09-13T09:21:44.105 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 217 k 2024-09-13T09:21:44.105 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T09:21:44.321 INFO:teuthology.orchestra.run.smithi122.stdout:cephadm-18.2.4-820.g48a7de7e.el9.noarch.rpm 1.0 MB/s | 226 kB 00:00 2024-09-13T09:21:44.322 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:21:44.322 INFO:teuthology.orchestra.run.smithi122.stdout:Total 1.0 MB/s | 226 kB 00:00 2024-09-13T09:21:44.322 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T09:21:44.327 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T09:21:44.327 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T09:21:44.330 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T09:21:44.330 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T09:21:44.378 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-820.g48a7de7e.el9.n 4/5 2024-09-13T09:21:44.404 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T09:21:44.411 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-820.g48a7de7e.el9.noarch 5/5 2024-09-13T09:21:44.817 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:18.2.4-820.g48a7de7e.el9.noarch 1/1 2024-09-13T09:21:44.899 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephadm-2:18.2.4-820.g48a7de7e.el9.noarch 1/1 2024-09-13T09:21:45.224 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:18.2.4-820.g48a7de7e.el9.noarch 1/1 2024-09-13T09:21:45.483 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-820.g48a7de7e.el9.noarch 5/5 2024-09-13T09:21:45.484 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-820.g48a7de7e.el9.n 1/5 2024-09-13T09:21:45.484 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-820.g48a7de7e.el9.noarch 2/5 2024-09-13T09:21:45.484 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-820.g48a7de7e.el9.no 3/5 2024-09-13T09:21:45.484 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-09-13T09:21:45.486 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephadm-2:18.2.4-820.g48a7de7e.el9.noarch 1/1 2024-09-13T09:21:45.486 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:45.487 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T09:21:45.487 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm-2:18.2.4-820.g48a7de7e.el9.noarch 2024-09-13T09:21:45.487 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:45.487 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:21:45.702 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T09:21:45.744 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-09-13T09:21:45.744 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:45.744 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-13T09:21:45.744 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards-2:18.2.4-820.g48a7de7e.el9.noarch 2024-09-13T09:21:45.745 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard-2:18.2.4-820.g48a7de7e.el9.noarch 2024-09-13T09:21:45.745 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts-2:18.2.4-820.g48a7de7e.el9.noarch 2024-09-13T09:21:45.745 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T09:21:45.745 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T09:21:45.745 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:45.745 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:21:45.968 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T09:21:46.279 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:52 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:21:46.411 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:21:46.412 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:21:46.412 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-09-13T09:21:46.412 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:21:46.412 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T09:21:46.412 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 143 k 2024-09-13T09:21:46.412 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:46.412 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T09:21:46.413 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:21:46.413 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T09:21:46.413 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:46.413 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 143 k 2024-09-13T09:21:46.413 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 395 k 2024-09-13T09:21:46.413 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T09:21:46.523 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:00 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:21:46.629 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-immutable-object-cache-18.2.4-820.g48a7de7 660 kB/s | 143 kB 00:00 2024-09-13T09:21:46.630 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:21:46.630 INFO:teuthology.orchestra.run.smithi122.stdout:Total 655 kB/s | 143 kB 00:00 2024-09-13T09:21:46.630 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T09:21:46.637 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T09:21:46.638 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T09:21:46.656 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:21:46.657 INFO:teuthology.orchestra.run.smithi190.stdout:====================================================================================== 2024-09-13T09:21:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-13T09:21:46.657 INFO:teuthology.orchestra.run.smithi190.stdout:====================================================================================== 2024-09-13T09:21:46.657 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-13T09:21:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-820.g48a7de7e.el9 ceph-noarch 7.4 M 2024-09-13T09:21:46.657 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-13T09:21:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T09:21:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T09:21:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T09:21:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T09:21:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T09:21:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T09:21:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T09:21:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T09:21:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T09:21:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T09:21:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T09:21:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:46.658 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-13T09:21:46.658 INFO:teuthology.orchestra.run.smithi190.stdout:====================================================================================== 2024-09-13T09:21:46.658 INFO:teuthology.orchestra.run.smithi190.stdout:Install 12 Packages 2024-09-13T09:21:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:46.659 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 38 M 2024-09-13T09:21:46.659 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 212 M 2024-09-13T09:21:46.659 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-13T09:21:46.694 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T09:21:46.695 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T09:21:46.824 INFO:teuthology.orchestra.run.smithi190.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.6 MB/s | 191 kB 00:00 2024-09-13T09:21:46.834 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T09:21:46.889 INFO:teuthology.orchestra.run.smithi190.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.3 MB/s | 801 kB 00:00 2024-09-13T09:21:46.947 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-820.g48a7de7e.e 1/1 2024-09-13T09:21:46.973 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-820.g48a7de7e.e 1/1 2024-09-13T09:21:46.973 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T09:21:46.973 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T09:21:46.973 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:47.223 INFO:teuthology.orchestra.run.smithi190.stdout:(3/12): ceph-mgr-diskprediction-local-18.2.4-82 14 MB/s | 7.4 MB 00:00 2024-09-13T09:21:47.415 INFO:teuthology.orchestra.run.smithi190.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 55 kB/s | 33 kB 00:00 2024-09-13T09:21:47.699 INFO:teuthology.orchestra.run.smithi190.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 38 kB/s | 18 kB 00:00 2024-09-13T09:21:47.904 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-820.g48a7de7e.e 1/1 2024-09-13T09:21:47.904 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:47.904 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T09:21:47.904 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:21:47.904 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:47.904 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:21:47.974 INFO:teuthology.orchestra.run.smithi190.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.8 MB/s | 3.0 MB 00:01 2024-09-13T09:21:47.999 INFO:teuthology.orchestra.run.smithi190.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 66 kB/s | 38 kB 00:00 2024-09-13T09:21:48.137 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr 2024-09-13T09:21:48.701 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:54 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:21:48.725 INFO:teuthology.orchestra.run.smithi190.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 326 kB/s | 245 kB 00:00 2024-09-13T09:21:48.796 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-mgr-2:18.2.4-820.g48a7de7e.el9.x86_64 is already installed. 2024-09-13T09:21:48.825 INFO:teuthology.orchestra.run.smithi190.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 4.4 MB/s | 4.9 MB 00:01 2024-09-13T09:21:48.838 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:21:48.839 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T09:21:48.840 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:21:48.915 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T09:21:49.159 INFO:teuthology.orchestra.run.smithi190.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.4 MB/s | 5.1 MB 00:01 2024-09-13T09:21:49.480 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:55 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:21:49.600 INFO:teuthology.orchestra.run.smithi190.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 286 kB/s | 250 kB 00:00 2024-09-13T09:21:49.615 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:21:49.615 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:21:49.616 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T09:21:49.616 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:21:49.616 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T09:21:49.616 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard noarch 2:18.2.4-820.g48a7de7e.el9 ceph-noarch 3.5 M 2024-09-13T09:21:49.616 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T09:21:49.616 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards noarch 2:18.2.4-820.g48a7de7e.el9 ceph-noarch 24 k 2024-09-13T09:21:49.616 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts noarch 2:18.2.4-820.g48a7de7e.el9 ceph-noarch 16 k 2024-09-13T09:21:49.616 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T09:21:49.616 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T09:21:49.616 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:49.616 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T09:21:49.616 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:21:49.616 INFO:teuthology.orchestra.run.smithi122.stdout:Install 5 Packages 2024-09-13T09:21:49.617 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:49.617 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 3.8 M 2024-09-13T09:21:49.617 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 87 M 2024-09-13T09:21:49.617 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T09:21:49.792 INFO:teuthology.orchestra.run.smithi122.stdout:(1/5): ceph-grafana-dashboards-18.2.4-820.g48a7 164 kB/s | 24 kB 00:00 2024-09-13T09:21:49.817 INFO:teuthology.orchestra.run.smithi122.stdout:(2/5): ceph-prometheus-alerts-18.2.4-820.g48a7d 95 kB/s | 16 kB 00:00 2024-09-13T09:21:49.859 INFO:teuthology.orchestra.run.smithi122.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 459 kB/s | 31 kB 00:00 2024-09-13T09:21:49.885 INFO:teuthology.orchestra.run.smithi122.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-09-13T09:21:50.043 INFO:teuthology.orchestra.run.smithi122.stdout:(5/5): ceph-mgr-dashboard-18.2.4-820.g48a7de7e. 8.8 MB/s | 3.5 MB 00:00 2024-09-13T09:21:50.046 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:21:50.047 INFO:teuthology.orchestra.run.smithi122.stdout:Total 8.8 MB/s | 3.8 MB 00:00 2024-09-13T09:21:50.047 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T09:21:50.054 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T09:21:50.054 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T09:21:50.137 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T09:21:50.137 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T09:21:50.292 INFO:teuthology.orchestra.run.smithi190.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-13T09:21:50.294 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:21:50.294 INFO:teuthology.orchestra.run.smithi190.stdout:Total 11 MB/s | 38 MB 00:03 2024-09-13T09:21:50.400 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T09:21:50.491 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-09-13T09:21:50.531 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-09-13T09:21:50.560 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-820.g48a7de7e.el9.no 3/5 2024-09-13T09:21:50.650 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-13T09:21:50.676 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-13T09:21:50.676 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-13T09:21:51.175 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-13T09:21:51.175 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-13T09:21:51.612 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-820.g48a7de7e.el9.n 4/5 2024-09-13T09:21:51.656 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-820.g48a7de7e.el9.noarch 5/5 2024-09-13T09:21:51.964 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-13T09:21:52.055 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T09:21:52.099 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T09:21:52.207 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T09:21:52.250 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T09:21:52.434 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T09:21:52.465 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T09:21:52.544 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T09:21:52.811 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-820.g48a7de7e.el9.noarch 5/5 2024-09-13T09:21:52.811 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-820.g48a7de7e.el9.n 1/5 2024-09-13T09:21:52.811 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-820.g48a7de7e.el9.noarch 2/5 2024-09-13T09:21:52.811 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-820.g48a7de7e.el9.no 3/5 2024-09-13T09:21:52.811 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-09-13T09:21:53.061 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-09-13T09:21:53.061 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:53.061 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T09:21:53.061 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards-2:18.2.4-820.g48a7de7e.el9.noarch 2024-09-13T09:21:53.061 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard-2:18.2.4-820.g48a7de7e.el9.noarch 2024-09-13T09:21:53.061 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts-2:18.2.4-820.g48a7de7e.el9.noarch 2024-09-13T09:21:53.061 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T09:21:53.062 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T09:21:53.062 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:53.062 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:21:53.194 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T09:21:53.287 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T09:21:53.325 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T09:21:53.851 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:59 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:21:53.983 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:21:53.985 INFO:teuthology.orchestra.run.smithi122.stdout:====================================================================================== 2024-09-13T09:21:53.985 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T09:21:53.985 INFO:teuthology.orchestra.run.smithi122.stdout:====================================================================================== 2024-09-13T09:21:53.985 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T09:21:53.985 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-820.g48a7de7e.el9 ceph-noarch 7.4 M 2024-09-13T09:21:53.985 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T09:21:53.985 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T09:21:53.985 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T09:21:53.985 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T09:21:53.985 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T09:21:53.986 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T09:21:53.986 INFO:teuthology.orchestra.run.smithi122.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T09:21:53.986 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T09:21:53.986 INFO:teuthology.orchestra.run.smithi122.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T09:21:53.986 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T09:21:53.986 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T09:21:53.986 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T09:21:53.986 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:53.986 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T09:21:53.986 INFO:teuthology.orchestra.run.smithi122.stdout:====================================================================================== 2024-09-13T09:21:53.986 INFO:teuthology.orchestra.run.smithi122.stdout:Install 12 Packages 2024-09-13T09:21:53.986 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:21:53.987 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 38 M 2024-09-13T09:21:53.987 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 212 M 2024-09-13T09:21:53.987 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T09:21:54.451 INFO:teuthology.orchestra.run.smithi122.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-13T09:21:54.501 INFO:teuthology.orchestra.run.smithi122.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.1 MB/s | 801 kB 00:00 2024-09-13T09:21:54.568 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T09:21:54.593 INFO:teuthology.orchestra.run.smithi122.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 230 kB/s | 33 kB 00:00 2024-09-13T09:21:54.643 INFO:teuthology.orchestra.run.smithi122.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 366 kB/s | 18 kB 00:00 2024-09-13T09:21:54.718 INFO:teuthology.orchestra.run.smithi122.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 512 kB/s | 38 kB 00:00 2024-09-13T09:21:54.810 INFO:teuthology.orchestra.run.smithi122.stdout:(6/12): ceph-mgr-diskprediction-local-18.2.4-82 15 MB/s | 7.4 MB 00:00 2024-09-13T09:21:54.977 INFO:teuthology.orchestra.run.smithi122.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-09-13T09:21:55.277 INFO:teuthology.orchestra.run.smithi122.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.9 MB/s | 3.0 MB 00:00 2024-09-13T09:21:55.361 INFO:teuthology.orchestra.run.smithi122.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-09-13T09:21:55.619 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T09:21:55.657 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-820.g48a7de 12/12 2024-09-13T09:21:55.886 INFO:teuthology.orchestra.run.smithi122.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.2 MB/s | 4.9 MB 00:01 2024-09-13T09:21:56.070 INFO:teuthology.orchestra.run.smithi122.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.7 MB/s | 5.1 MB 00:01 2024-09-13T09:21:57.686 INFO:teuthology.orchestra.run.smithi122.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 7.0 MB/s | 16 MB 00:02 2024-09-13T09:21:57.687 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:21:57.688 INFO:teuthology.orchestra.run.smithi122.stdout:Total 10 MB/s | 38 MB 00:03 2024-09-13T09:21:57.996 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-820.g48a7de 12/12 2024-09-13T09:21:57.996 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-820.g48a7de 1/12 2024-09-13T09:21:57.996 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T09:21:57.996 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T09:21:57.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T09:21:57.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T09:21:57.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T09:21:57.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T09:21:57.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T09:21:57.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T09:21:57.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T09:21:57.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T09:21:58.055 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T09:21:58.082 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T09:21:58.082 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T09:21:58.448 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T09:21:58.448 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:58.448 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-13T09:21:58.448 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local-2:18.2.4-820.g48a7de7e.el9.noarch 2024-09-13T09:21:58.448 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T09:21:58.448 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T09:21:58.448 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T09:21:58.448 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T09:21:58.448 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T09:21:58.449 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T09:21:58.449 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T09:21:58.449 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T09:21:58.449 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T09:21:58.449 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T09:21:58.449 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T09:21:58.449 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:58.449 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:21:58.593 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T09:21:58.593 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T09:21:58.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-rook 2024-09-13T09:21:59.373 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T09:21:59.413 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:13 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:21:59.453 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T09:21:59.493 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T09:21:59.551 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:21:59.552 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:21:59.552 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-13T09:21:59.552 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:21:59.552 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-13T09:21:59.552 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook noarch 2:18.2.4-820.g48a7de7e.el9 ceph-noarch 49 k 2024-09-13T09:21:59.552 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-13T09:21:59.553 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T09:21:59.553 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T09:21:59.553 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T09:21:59.553 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T09:21:59.553 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T09:21:59.553 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T09:21:59.553 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T09:21:59.553 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T09:21:59.553 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T09:21:59.553 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T09:21:59.553 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T09:21:59.553 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T09:21:59.553 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:59.553 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-13T09:21:59.554 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:21:59.554 INFO:teuthology.orchestra.run.smithi190.stdout:Install 13 Packages 2024-09-13T09:21:59.554 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:21:59.554 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 2.2 M 2024-09-13T09:21:59.554 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 26 M 2024-09-13T09:21:59.554 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-13T09:21:59.591 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T09:21:59.636 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T09:21:59.822 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T09:21:59.860 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T09:21:59.909 INFO:teuthology.orchestra.run.smithi190.stdout:(1/13): ceph-mgr-rook-18.2.4-820.g48a7de7e.el9. 295 kB/s | 49 kB 00:00 2024-09-13T09:21:59.937 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T09:22:00.026 INFO:teuthology.orchestra.run.smithi190.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 67 kB/s | 19 kB 00:00 2024-09-13T09:22:00.052 INFO:teuthology.orchestra.run.smithi190.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 83 kB/s | 26 kB 00:00 2024-09-13T09:22:00.402 INFO:teuthology.orchestra.run.smithi190.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 423 kB/s | 159 kB 00:00 2024-09-13T09:22:00.519 INFO:teuthology.orchestra.run.smithi190.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 597 kB/s | 279 kB 00:00 2024-09-13T09:22:00.544 INFO:teuthology.orchestra.run.smithi190.stdout:(6/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 350 kB/s | 222 kB 00:00 2024-09-13T09:22:00.570 INFO:teuthology.orchestra.run.smithi190.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 320 kB/s | 54 kB 00:00 2024-09-13T09:22:00.598 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T09:22:00.619 INFO:teuthology.orchestra.run.smithi190.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 321 kB/s | 32 kB 00:00 2024-09-13T09:22:00.645 INFO:teuthology.orchestra.run.smithi190.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 188 kB/s | 14 kB 00:00 2024-09-13T09:22:00.679 INFO:teuthology.orchestra.run.smithi190.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-09-13T09:22:00.712 INFO:teuthology.orchestra.run.smithi190.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.5 MB/s | 220 kB 00:00 2024-09-13T09:22:00.722 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T09:22:00.737 INFO:teuthology.orchestra.run.smithi190.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-09-13T09:22:00.780 INFO:teuthology.orchestra.run.smithi190.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.5 MB/s | 1.0 MB 00:00 2024-09-13T09:22:00.782 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:22:00.782 INFO:teuthology.orchestra.run.smithi190.stdout:Total 1.8 MB/s | 2.2 MB 00:01 2024-09-13T09:22:00.845 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-13T09:22:00.860 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-13T09:22:00.860 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-13T09:22:00.949 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-13T09:22:00.949 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-13T09:22:01.472 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-13T09:22:01.642 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T09:22:01.714 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T09:22:01.770 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T09:22:01.807 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T09:22:01.856 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T09:22:01.968 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T09:22:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T09:22:02.069 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T09:22:02.116 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T09:22:02.832 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T09:22:02.881 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T09:22:02.915 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T09:22:02.962 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T09:22:02.984 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-rook-2:18.2.4-820.g48a7de7e.el9.noarch 13/13 2024-09-13T09:22:03.006 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T09:22:03.034 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-820.g48a7de 12/12 2024-09-13T09:22:03.748 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-820.g48a7de7e.el9.noarch 13/13 2024-09-13T09:22:03.749 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-rook-2:18.2.4-820.g48a7de7e.el9.noarch 1/13 2024-09-13T09:22:03.749 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T09:22:03.749 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T09:22:03.749 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T09:22:03.749 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T09:22:03.749 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T09:22:03.749 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T09:22:03.749 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T09:22:03.749 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T09:22:03.749 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T09:22:03.749 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T09:22:03.749 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T09:22:04.076 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T09:22:04.077 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:04.077 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-13T09:22:04.077 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook-2:18.2.4-820.g48a7de7e.el9.noarch 2024-09-13T09:22:04.077 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T09:22:04.077 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T09:22:04.077 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T09:22:04.077 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T09:22:04.077 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T09:22:04.077 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T09:22:04.077 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T09:22:04.077 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T09:22:04.078 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T09:22:04.078 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T09:22:04.078 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T09:22:04.078 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T09:22:04.078 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:04.078 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:22:04.342 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T09:22:04.907 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:18 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:22:05.043 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:22:05.043 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:05.043 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-13T09:22:05.044 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:05.044 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-13T09:22:05.044 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm noarch 2:18.2.4-820.g48a7de7e.el9 ceph-noarch 139 k 2024-09-13T09:22:05.044 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-13T09:22:05.044 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T09:22:05.044 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T09:22:05.044 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T09:22:05.044 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T09:22:05.044 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T09:22:05.044 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:05.044 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-13T09:22:05.044 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:05.044 INFO:teuthology.orchestra.run.smithi190.stdout:Install 6 Packages 2024-09-13T09:22:05.045 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:05.045 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 7.0 M 2024-09-13T09:22:05.045 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 33 M 2024-09-13T09:22:05.045 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-13T09:22:05.491 INFO:teuthology.orchestra.run.smithi190.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-09-13T09:22:05.541 INFO:teuthology.orchestra.run.smithi190.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-13T09:22:05.591 INFO:teuthology.orchestra.run.smithi190.stdout:(3/6): ceph-mgr-cephadm-18.2.4-820.g48a7de7e.el 571 kB/s | 139 kB 00:00 2024-09-13T09:22:05.594 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-820.g48a7de 12/12 2024-09-13T09:22:05.594 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-820.g48a7de 1/12 2024-09-13T09:22:05.594 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T09:22:05.594 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T09:22:05.595 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T09:22:05.595 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T09:22:05.595 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T09:22:05.595 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T09:22:05.595 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T09:22:05.595 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T09:22:05.595 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T09:22:05.595 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T09:22:05.608 INFO:teuthology.orchestra.run.smithi190.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 859 kB/s | 58 kB 00:00 2024-09-13T09:22:05.658 INFO:teuthology.orchestra.run.smithi190.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-13T09:22:05.742 INFO:teuthology.orchestra.run.smithi190.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-09-13T09:22:05.745 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:22:05.745 INFO:teuthology.orchestra.run.smithi190.stdout:Total 10 MB/s | 7.0 MB 00:00 2024-09-13T09:22:05.814 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-13T09:22:05.825 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-13T09:22:05.825 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-13T09:22:05.924 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-13T09:22:05.924 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-13T09:22:06.003 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T09:22:06.004 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:06.004 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T09:22:06.004 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local-2:18.2.4-820.g48a7de7e.el9.noarch 2024-09-13T09:22:06.004 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T09:22:06.004 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T09:22:06.004 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T09:22:06.004 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T09:22:06.004 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T09:22:06.004 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T09:22:06.004 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T09:22:06.005 INFO:teuthology.orchestra.run.smithi122.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T09:22:06.005 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T09:22:06.005 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T09:22:06.005 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T09:22:06.005 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:06.008 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:22:06.226 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-13T09:22:06.381 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T09:22:06.428 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T09:22:06.448 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-rook 2024-09-13T09:22:06.964 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T09:22:07.021 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:13 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:22:07.055 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T09:22:07.116 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T09:22:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-820.g48a7de7e.el9.noarch 6/6 2024-09-13T09:22:07.153 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:22:07.154 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:07.155 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T09:22:07.155 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:07.155 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T09:22:07.155 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook noarch 2:18.2.4-820.g48a7de7e.el9 ceph-noarch 49 k 2024-09-13T09:22:07.155 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T09:22:07.155 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T09:22:07.155 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T09:22:07.155 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T09:22:07.155 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T09:22:07.155 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T09:22:07.155 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T09:22:07.155 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T09:22:07.155 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T09:22:07.156 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T09:22:07.156 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T09:22:07.156 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T09:22:07.156 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T09:22:07.156 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:07.156 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T09:22:07.156 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:07.156 INFO:teuthology.orchestra.run.smithi122.stdout:Install 13 Packages 2024-09-13T09:22:07.156 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:07.157 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 2.2 M 2024-09-13T09:22:07.157 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 26 M 2024-09-13T09:22:07.157 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T09:22:07.559 INFO:teuthology.orchestra.run.smithi122.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 318 kB/s | 26 kB 00:00 2024-09-13T09:22:07.585 INFO:teuthology.orchestra.run.smithi122.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 181 kB/s | 19 kB 00:00 2024-09-13T09:22:07.660 INFO:teuthology.orchestra.run.smithi122.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 222 kB 00:00 2024-09-13T09:22:07.686 INFO:teuthology.orchestra.run.smithi122.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 159 kB 00:00 2024-09-13T09:22:07.711 INFO:teuthology.orchestra.run.smithi122.stdout:(5/13): ceph-mgr-rook-18.2.4-820.g48a7de7e.el9. 213 kB/s | 49 kB 00:00 2024-09-13T09:22:07.736 INFO:teuthology.orchestra.run.smithi122.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-13T09:22:07.761 INFO:teuthology.orchestra.run.smithi122.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 279 kB 00:00 2024-09-13T09:22:07.812 INFO:teuthology.orchestra.run.smithi122.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 320 kB/s | 32 kB 00:00 2024-09-13T09:22:07.837 INFO:teuthology.orchestra.run.smithi122.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-13T09:22:07.870 INFO:teuthology.orchestra.run.smithi122.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-09-13T09:22:07.896 INFO:teuthology.orchestra.run.smithi122.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.8 MB/s | 220 kB 00:00 2024-09-13T09:22:07.921 INFO:teuthology.orchestra.run.smithi122.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-13T09:22:07.953 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-820.g48a7de7e.el9.noarch 6/6 2024-09-13T09:22:07.953 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-820.g48a7de7e.el9.noarch 1/6 2024-09-13T09:22:07.953 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T09:22:07.953 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T09:22:07.953 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T09:22:07.953 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T09:22:07.954 INFO:teuthology.orchestra.run.smithi122.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.3 MB/s | 1.0 MB 00:00 2024-09-13T09:22:07.957 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:22:07.957 INFO:teuthology.orchestra.run.smithi122.stdout:Total 2.8 MB/s | 2.2 MB 00:00 2024-09-13T09:22:08.019 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T09:22:08.034 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T09:22:08.034 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T09:22:08.125 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T09:22:08.125 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T09:22:08.226 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T09:22:08.226 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:08.226 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-13T09:22:08.226 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm-2:18.2.4-820.g48a7de7e.el9.noarch 2024-09-13T09:22:08.226 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T09:22:08.226 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T09:22:08.226 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T09:22:08.226 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T09:22:08.226 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T09:22:08.227 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:08.227 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:22:08.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-fuse 2024-09-13T09:22:08.642 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T09:22:08.805 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T09:22:08.873 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T09:22:08.929 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T09:22:08.965 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T09:22:09.007 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T09:22:09.061 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:23 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:22:09.110 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T09:22:09.196 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:22:09.196 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:09.196 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-13T09:22:09.196 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:09.197 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-13T09:22:09.197 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 852 k 2024-09-13T09:22:09.197 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-13T09:22:09.197 INFO:teuthology.orchestra.run.smithi190.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T09:22:09.197 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:09.197 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-13T09:22:09.197 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:09.197 INFO:teuthology.orchestra.run.smithi190.stdout:Install 2 Packages 2024-09-13T09:22:09.197 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:09.198 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 932 k 2024-09-13T09:22:09.198 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 2.7 M 2024-09-13T09:22:09.198 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-13T09:22:09.228 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T09:22:09.284 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T09:22:09.595 INFO:teuthology.orchestra.run.smithi190.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 662 kB/s | 80 kB 00:00 2024-09-13T09:22:09.778 INFO:teuthology.orchestra.run.smithi190.stdout:(2/2): ceph-fuse-18.2.4-820.g48a7de7e.el9.x86_6 2.7 MB/s | 852 kB 00:00 2024-09-13T09:22:09.779 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:22:09.779 INFO:teuthology.orchestra.run.smithi190.stdout:Total 1.6 MB/s | 932 kB 00:00 2024-09-13T09:22:09.791 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-13T09:22:09.813 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-13T09:22:09.813 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-13T09:22:09.892 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-13T09:22:09.893 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-13T09:22:09.996 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T09:22:10.056 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T09:22:10.075 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-13T09:22:10.089 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T09:22:10.137 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T09:22:10.159 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-rook-2:18.2.4-820.g48a7de7e.el9.noarch 13/13 2024-09-13T09:22:10.244 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T09:22:10.312 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-fuse-2:18.2.4-820.g48a7de7e.el9.x86_64 2/2 2024-09-13T09:22:10.945 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-820.g48a7de7e.el9.noarch 13/13 2024-09-13T09:22:10.945 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-rook-2:18.2.4-820.g48a7de7e.el9.noarch 1/13 2024-09-13T09:22:10.945 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T09:22:10.945 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T09:22:10.946 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T09:22:10.946 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T09:22:10.946 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T09:22:10.946 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T09:22:10.946 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T09:22:10.946 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T09:22:10.946 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T09:22:10.946 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T09:22:10.946 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T09:22:11.004 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-fuse-2:18.2.4-820.g48a7de7e.el9.x86_64 2/2 2024-09-13T09:22:11.004 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-fuse-2:18.2.4-820.g48a7de7e.el9.x86_64 1/2 2024-09-13T09:22:11.268 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T09:22:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:11.269 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T09:22:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook-2:18.2.4-820.g48a7de7e.el9.noarch 2024-09-13T09:22:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T09:22:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T09:22:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T09:22:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T09:22:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T09:22:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T09:22:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T09:22:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T09:22:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T09:22:11.270 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T09:22:11.270 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T09:22:11.270 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T09:22:11.270 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:11.270 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:22:11.285 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T09:22:11.285 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:11.285 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-13T09:22:11.285 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse-2:18.2.4-820.g48a7de7e.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T09:22:11.285 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:11.285 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:22:11.548 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T09:22:11.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-volume 2024-09-13T09:22:12.119 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:18 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:22:12.134 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:26 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:22:12.252 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:22:12.253 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:12.253 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T09:22:12.253 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:12.253 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T09:22:12.253 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm noarch 2:18.2.4-820.g48a7de7e.el9 ceph-noarch 139 k 2024-09-13T09:22:12.253 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T09:22:12.253 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T09:22:12.253 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T09:22:12.253 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T09:22:12.254 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T09:22:12.254 INFO:teuthology.orchestra.run.smithi122.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T09:22:12.254 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:12.254 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T09:22:12.254 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:12.254 INFO:teuthology.orchestra.run.smithi122.stdout:Install 6 Packages 2024-09-13T09:22:12.254 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:12.254 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 7.0 M 2024-09-13T09:22:12.255 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 33 M 2024-09-13T09:22:12.255 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T09:22:12.270 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:22:12.271 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:12.271 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-13T09:22:12.271 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:12.271 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-13T09:22:12.271 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume noarch 2:18.2.4-820.g48a7de7e.el9 ceph-noarch 264 k 2024-09-13T09:22:12.271 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-13T09:22:12.271 INFO:teuthology.orchestra.run.smithi190.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T09:22:12.271 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:12.271 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-13T09:22:12.271 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:12.272 INFO:teuthology.orchestra.run.smithi190.stdout:Install 2 Packages 2024-09-13T09:22:12.272 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:12.272 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 342 k 2024-09-13T09:22:12.272 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 1.5 M 2024-09-13T09:22:12.272 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-13T09:22:12.607 INFO:teuthology.orchestra.run.smithi122.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-13T09:22:12.632 INFO:teuthology.orchestra.run.smithi122.stdout:(2/6): ceph-mgr-cephadm-18.2.4-820.g48a7de7e.el 595 kB/s | 139 kB 00:00 2024-09-13T09:22:12.683 INFO:teuthology.orchestra.run.smithi122.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-09-13T09:22:12.708 INFO:teuthology.orchestra.run.smithi122.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 766 kB/s | 58 kB 00:00 2024-09-13T09:22:12.733 INFO:teuthology.orchestra.run.smithi122.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-13T09:22:12.734 INFO:teuthology.orchestra.run.smithi190.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 423 kB/s | 77 kB 00:00 2024-09-13T09:22:12.776 INFO:teuthology.orchestra.run.smithi190.stdout:(2/2): ceph-volume-18.2.4-820.g48a7de7e.el9.noa 1.2 MB/s | 264 kB 00:00 2024-09-13T09:22:12.776 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:22:12.777 INFO:teuthology.orchestra.run.smithi190.stdout:Total 677 kB/s | 342 kB 00:00 2024-09-13T09:22:12.788 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-13T09:22:12.798 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-13T09:22:12.798 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-13T09:22:12.831 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-13T09:22:12.831 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-13T09:22:12.978 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-13T09:22:13.168 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-09-13T09:22:13.237 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-volume-2:18.2.4-820.g48a7de7e.el9.noarch 2/2 2024-09-13T09:22:13.253 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-volume-2:18.2.4-820.g48a7de7e.el9.noarch 2/2 2024-09-13T09:22:13.253 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T09:22:13.253 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T09:22:13.253 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:13.634 INFO:teuthology.orchestra.run.smithi122.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.8 MB/s | 6.0 MB 00:01 2024-09-13T09:22:13.636 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:22:13.636 INFO:teuthology.orchestra.run.smithi122.stdout:Total 5.1 MB/s | 7.0 MB 00:01 2024-09-13T09:22:13.707 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T09:22:13.718 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T09:22:13.718 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T09:22:13.818 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T09:22:13.819 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T09:22:13.984 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-volume-2:18.2.4-820.g48a7de7e.el9.noarch 1/2 2024-09-13T09:22:14.122 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T09:22:14.291 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T09:22:14.311 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-09-13T09:22:14.311 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:14.311 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-13T09:22:14.312 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume-2:18.2.4-820.g48a7de7e.el9.noarch 2024-09-13T09:22:14.312 INFO:teuthology.orchestra.run.smithi190.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T09:22:14.312 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:14.312 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:22:14.343 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T09:22:14.581 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados-devel 2024-09-13T09:22:14.865 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T09:22:14.947 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T09:22:15.009 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T09:22:15.048 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-820.g48a7de7e.el9.noarch 6/6 2024-09-13T09:22:15.165 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:29 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:22:15.300 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:22:15.300 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:15.300 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-09-13T09:22:15.300 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:15.300 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-13T09:22:15.300 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 127 k 2024-09-13T09:22:15.300 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:15.300 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-13T09:22:15.301 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:15.301 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-13T09:22:15.301 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:15.301 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 127 k 2024-09-13T09:22:15.301 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 456 k 2024-09-13T09:22:15.301 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-13T09:22:15.515 INFO:teuthology.orchestra.run.smithi190.stdout:librados-devel-18.2.4-820.g48a7de7e.el9.x86_64. 592 kB/s | 127 kB 00:00 2024-09-13T09:22:15.516 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:22:15.516 INFO:teuthology.orchestra.run.smithi190.stdout:Total 588 kB/s | 127 kB 00:00 2024-09-13T09:22:15.516 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-13T09:22:15.522 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-13T09:22:15.522 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-13T09:22:15.588 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-13T09:22:15.588 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-13T09:22:15.734 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-13T09:22:15.844 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-820.g48a7de7e.el9.noarch 6/6 2024-09-13T09:22:15.845 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-820.g48a7de7e.el9.noarch 1/6 2024-09-13T09:22:15.845 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T09:22:15.845 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T09:22:15.845 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T09:22:15.845 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T09:22:15.879 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librados-devel-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T09:22:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:16.108 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T09:22:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm-2:18.2.4-820.g48a7de7e.el9.noarch 2024-09-13T09:22:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T09:22:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T09:22:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T09:22:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T09:22:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T09:22:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:16.109 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:22:16.301 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados-devel-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:16.366 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-fuse 2024-09-13T09:22:16.577 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados-devel-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:16.577 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:16.577 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-13T09:22:16.577 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:22:16.577 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:16.577 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:22:16.811 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs2 2024-09-13T09:22:16.937 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:22 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:22:17.069 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:22:17.070 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:17.070 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T09:22:17.070 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:17.071 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T09:22:17.071 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 852 k 2024-09-13T09:22:17.071 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T09:22:17.071 INFO:teuthology.orchestra.run.smithi122.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T09:22:17.071 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:17.071 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T09:22:17.071 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:17.071 INFO:teuthology.orchestra.run.smithi122.stdout:Install 2 Packages 2024-09-13T09:22:17.071 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:17.071 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 932 k 2024-09-13T09:22:17.071 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 2.7 M 2024-09-13T09:22:17.071 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T09:22:17.302 INFO:teuthology.orchestra.run.smithi122.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 431 kB/s | 80 kB 00:00 2024-09-13T09:22:17.385 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:31 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:22:17.402 INFO:teuthology.orchestra.run.smithi122.stdout:(2/2): ceph-fuse-18.2.4-820.g48a7de7e.el9.x86_6 2.9 MB/s | 852 kB 00:00 2024-09-13T09:22:17.403 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:22:17.403 INFO:teuthology.orchestra.run.smithi122.stdout:Total 2.7 MB/s | 932 kB 00:00 2024-09-13T09:22:17.415 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T09:22:17.437 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T09:22:17.437 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T09:22:17.476 INFO:teuthology.orchestra.run.smithi190.stdout:Package libcephfs2-2:18.2.4-820.g48a7de7e.el9.x86_64 is already installed. 2024-09-13T09:22:17.518 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T09:22:17.519 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T09:22:17.520 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:22:17.521 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-13T09:22:17.521 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:22:17.589 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs-devel 2024-09-13T09:22:17.704 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T09:22:17.843 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T09:22:17.919 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-fuse-2:18.2.4-820.g48a7de7e.el9.x86_64 2/2 2024-09-13T09:22:18.157 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:32 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:22:18.289 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:22:18.290 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:18.290 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-09-13T09:22:18.291 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:18.291 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-13T09:22:18.291 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 31 k 2024-09-13T09:22:18.291 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:18.291 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-13T09:22:18.291 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:18.291 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-13T09:22:18.291 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:18.291 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 31 k 2024-09-13T09:22:18.291 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 139 k 2024-09-13T09:22:18.291 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-13T09:22:18.437 INFO:teuthology.orchestra.run.smithi190.stdout:libcephfs-devel-18.2.4-820.g48a7de7e.el9.x86_64 214 kB/s | 31 kB 00:00 2024-09-13T09:22:18.437 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:22:18.437 INFO:teuthology.orchestra.run.smithi190.stdout:Total 212 kB/s | 31 kB 00:00 2024-09-13T09:22:18.437 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-13T09:22:18.441 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-13T09:22:18.441 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-13T09:22:18.458 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-13T09:22:18.458 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-13T09:22:18.540 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-13T09:22:18.618 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-fuse-2:18.2.4-820.g48a7de7e.el9.x86_64 2/2 2024-09-13T09:22:18.618 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-fuse-2:18.2.4-820.g48a7de7e.el9.x86_64 1/2 2024-09-13T09:22:18.665 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs-devel-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:18.877 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T09:22:18.878 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:18.878 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T09:22:18.878 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse-2:18.2.4-820.g48a7de7e.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T09:22:18.878 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:18.878 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:22:19.042 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:19.172 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-volume 2024-09-13T09:22:19.290 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs-devel-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:19.290 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:19.290 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-13T09:22:19.290 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:22:19.290 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:19.290 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:22:19.463 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados2 2024-09-13T09:22:19.787 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:25 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:22:19.925 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:22:19.925 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:19.925 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T09:22:19.925 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:19.925 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T09:22:19.925 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume noarch 2:18.2.4-820.g48a7de7e.el9 ceph-noarch 264 k 2024-09-13T09:22:19.925 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T09:22:19.925 INFO:teuthology.orchestra.run.smithi122.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T09:22:19.926 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:19.926 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T09:22:19.926 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:19.926 INFO:teuthology.orchestra.run.smithi122.stdout:Install 2 Packages 2024-09-13T09:22:19.926 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:19.926 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 342 k 2024-09-13T09:22:19.926 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 1.5 M 2024-09-13T09:22:19.927 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T09:22:20.031 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:34 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:22:20.122 INFO:teuthology.orchestra.run.smithi190.stdout:Package librados2-2:18.2.4-820.g48a7de7e.el9.x86_64 is already installed. 2024-09-13T09:22:20.166 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:22:20.166 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-13T09:22:20.166 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:22:20.191 INFO:teuthology.orchestra.run.smithi122.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 536 kB/s | 77 kB 00:00 2024-09-13T09:22:20.231 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librbd1 2024-09-13T09:22:20.283 INFO:teuthology.orchestra.run.smithi122.stdout:(2/2): ceph-volume-18.2.4-820.g48a7de7e.el9.noa 1.1 MB/s | 264 kB 00:00 2024-09-13T09:22:20.284 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:22:20.284 INFO:teuthology.orchestra.run.smithi122.stdout:Total 954 kB/s | 342 kB 00:00 2024-09-13T09:22:20.296 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T09:22:20.306 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T09:22:20.306 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T09:22:20.338 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T09:22:20.338 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T09:22:20.483 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T09:22:20.675 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-09-13T09:22:20.719 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-volume-2:18.2.4-820.g48a7de7e.el9.noarch 2/2 2024-09-13T09:22:20.734 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-volume-2:18.2.4-820.g48a7de7e.el9.noarch 2/2 2024-09-13T09:22:20.735 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T09:22:20.735 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T09:22:20.735 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:20.801 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:34 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:22:20.894 INFO:teuthology.orchestra.run.smithi190.stdout:Package librbd1-2:18.2.4-820.g48a7de7e.el9.x86_64 is already installed. 2024-09-13T09:22:20.937 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:22:20.938 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-13T09:22:20.938 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:22:21.001 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rados 2024-09-13T09:22:21.481 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-volume-2:18.2.4-820.g48a7de7e.el9.noarch 1/2 2024-09-13T09:22:21.568 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:35 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:22:21.658 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rados-2:18.2.4-820.g48a7de7e.el9.x86_64 is already installed. 2024-09-13T09:22:21.701 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:22:21.702 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-13T09:22:21.702 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:22:21.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rgw 2024-09-13T09:22:21.775 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-09-13T09:22:21.775 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:21.775 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T09:22:21.775 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume-2:18.2.4-820.g48a7de7e.el9.noarch 2024-09-13T09:22:21.775 INFO:teuthology.orchestra.run.smithi122.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T09:22:21.775 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:21.776 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:22:22.090 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados-devel 2024-09-13T09:22:22.328 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:36 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:22:22.418 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rgw-2:18.2.4-820.g48a7de7e.el9.x86_64 is already installed. 2024-09-13T09:22:22.460 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:22:22.461 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-13T09:22:22.461 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:22:22.516 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-cephfs 2024-09-13T09:22:22.673 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:28 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:22:22.805 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:22:22.806 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:22.807 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-09-13T09:22:22.807 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:22.807 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T09:22:22.807 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 127 k 2024-09-13T09:22:22.807 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:22.807 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T09:22:22.807 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:22.808 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T09:22:22.808 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:22.808 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 127 k 2024-09-13T09:22:22.808 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 456 k 2024-09-13T09:22:22.808 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T09:22:22.989 INFO:teuthology.orchestra.run.smithi122.stdout:librados-devel-18.2.4-820.g48a7de7e.el9.x86_64. 697 kB/s | 127 kB 00:00 2024-09-13T09:22:22.990 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:22:22.990 INFO:teuthology.orchestra.run.smithi122.stdout:Total 691 kB/s | 127 kB 00:00 2024-09-13T09:22:22.990 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T09:22:22.995 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T09:22:22.996 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T09:22:23.059 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T09:22:23.060 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T09:22:23.083 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:37 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:22:23.174 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-cephfs-2:18.2.4-820.g48a7de7e.el9.x86_64 is already installed. 2024-09-13T09:22:23.214 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T09:22:23.217 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:22:23.218 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-13T09:22:23.218 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:22:23.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rbd 2024-09-13T09:22:23.313 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librados-devel-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:23.728 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados-devel-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:23.853 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:37 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:22:23.942 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rbd-2:18.2.4-820.g48a7de7e.el9.x86_64 is already installed. 2024-09-13T09:22:23.985 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:22:23.986 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-13T09:22:23.986 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:22:24.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados-devel-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:24.001 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:24.001 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T09:22:24.001 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:22:24.001 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:24.001 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:22:24.053 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-fuse 2024-09-13T09:22:24.220 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs2 2024-09-13T09:22:24.625 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:38 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:22:24.758 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:22:24.758 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:24.758 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-09-13T09:22:24.758 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:24.758 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-13T09:22:24.758 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 87 k 2024-09-13T09:22:24.759 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:24.759 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-13T09:22:24.759 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:24.759 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-13T09:22:24.759 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:24.760 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 87 k 2024-09-13T09:22:24.760 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 227 k 2024-09-13T09:22:24.760 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-13T09:22:24.788 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:30 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:22:24.878 INFO:teuthology.orchestra.run.smithi122.stdout:Package libcephfs2-2:18.2.4-820.g48a7de7e.el9.x86_64 is already installed. 2024-09-13T09:22:24.922 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:22:24.923 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T09:22:24.923 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:22:24.948 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-fuse-18.2.4-820.g48a7de7e.el9.x86_64.rpm 461 kB/s | 87 kB 00:00 2024-09-13T09:22:24.948 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:22:24.949 INFO:teuthology.orchestra.run.smithi190.stdout:Total 458 kB/s | 87 kB 00:00 2024-09-13T09:22:24.949 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-13T09:22:24.956 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-13T09:22:24.956 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-13T09:22:24.994 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs-devel 2024-09-13T09:22:25.020 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-13T09:22:25.021 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-13T09:22:25.159 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-13T09:22:25.262 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-fuse-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:25.552 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:31 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:22:25.654 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-fuse-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:25.685 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:22:25.685 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:25.685 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-09-13T09:22:25.685 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:25.685 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T09:22:25.685 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 31 k 2024-09-13T09:22:25.685 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:25.685 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T09:22:25.685 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:25.686 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T09:22:25.686 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:25.686 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 31 k 2024-09-13T09:22:25.686 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 139 k 2024-09-13T09:22:25.686 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T09:22:25.835 INFO:teuthology.orchestra.run.smithi122.stdout:libcephfs-devel-18.2.4-820.g48a7de7e.el9.x86_64 209 kB/s | 31 kB 00:00 2024-09-13T09:22:25.835 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:22:25.835 INFO:teuthology.orchestra.run.smithi122.stdout:Total 207 kB/s | 31 kB 00:00 2024-09-13T09:22:25.836 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T09:22:25.839 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T09:22:25.839 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T09:22:25.860 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T09:22:25.860 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T09:22:25.900 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-fuse-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:25.900 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:25.900 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-13T09:22:25.900 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:22:25.900 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:25.900 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:22:25.953 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T09:22:26.056 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs-devel-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:26.158 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-mirror 2024-09-13T09:22:26.411 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:26.639 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs-devel-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:26.639 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:26.639 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T09:22:26.640 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:22:26.640 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:26.640 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:22:26.733 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:40 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:22:26.849 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados2 2024-09-13T09:22:26.867 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:22:26.868 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:26.868 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-09-13T09:22:26.868 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:26.868 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-13T09:22:26.868 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 3.0 M 2024-09-13T09:22:26.868 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:26.868 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-13T09:22:26.868 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:26.869 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-13T09:22:26.869 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:26.869 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 3.0 M 2024-09-13T09:22:26.869 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 12 M 2024-09-13T09:22:26.869 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-13T09:22:27.194 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-mirror-18.2.4-820.g48a7de7e.el9.x86_64.rpm 9.2 MB/s | 3.0 MB 00:00 2024-09-13T09:22:27.194 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:22:27.195 INFO:teuthology.orchestra.run.smithi190.stdout:Total 9.2 MB/s | 3.0 MB 00:00 2024-09-13T09:22:27.195 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-13T09:22:27.204 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-13T09:22:27.204 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-13T09:22:27.284 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-13T09:22:27.284 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-13T09:22:27.416 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:33 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:22:27.506 INFO:teuthology.orchestra.run.smithi122.stdout:Package librados2-2:18.2.4-820.g48a7de7e.el9.x86_64 is already installed. 2024-09-13T09:22:27.548 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:22:27.549 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T09:22:27.549 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:22:27.618 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librbd1 2024-09-13T09:22:27.696 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-13T09:22:27.778 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-mirror-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:27.803 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-mirror-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:27.803 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T09:22:27.803 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T09:22:27.803 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T09:22:27.803 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T09:22:27.803 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:28.181 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:34 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:22:28.270 INFO:teuthology.orchestra.run.smithi122.stdout:Package librbd1-2:18.2.4-820.g48a7de7e.el9.x86_64 is already installed. 2024-09-13T09:22:28.312 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:22:28.313 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T09:22:28.313 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:22:28.387 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rados 2024-09-13T09:22:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-mirror-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:28.943 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-13T09:22:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:22:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:28.943 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:22:28.947 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:34 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:22:29.035 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rados-2:18.2.4-820.g48a7de7e.el9.x86_64 is already installed. 2024-09-13T09:22:29.078 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:22:29.079 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T09:22:29.079 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:22:29.150 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rgw 2024-09-13T09:22:29.168 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-nbd 2024-09-13T09:22:29.712 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:35 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:22:29.761 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:43 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:22:29.801 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rgw-2:18.2.4-820.g48a7de7e.el9.x86_64 is already installed. 2024-09-13T09:22:29.844 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:22:29.844 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T09:22:29.845 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:22:29.895 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:22:29.896 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:29.896 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-09-13T09:22:29.896 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:29.896 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-13T09:22:29.896 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 172 k 2024-09-13T09:22:29.896 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:29.896 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-13T09:22:29.896 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-13T09:22:29.897 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-13T09:22:29.897 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:29.897 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 172 k 2024-09-13T09:22:29.897 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 494 k 2024-09-13T09:22:29.897 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-13T09:22:29.920 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-cephfs 2024-09-13T09:22:30.107 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-nbd-18.2.4-820.g48a7de7e.el9.x86_64.rpm 819 kB/s | 172 kB 00:00 2024-09-13T09:22:30.107 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:22:30.108 INFO:teuthology.orchestra.run.smithi190.stdout:Total 813 kB/s | 172 kB 00:00 2024-09-13T09:22:30.108 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-13T09:22:30.115 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-13T09:22:30.116 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-13T09:22:30.179 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-13T09:22:30.179 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-13T09:22:30.335 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-13T09:22:30.429 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-nbd-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:30.478 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:36 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:22:30.567 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-cephfs-2:18.2.4-820.g48a7de7e.el9.x86_64 is already installed. 2024-09-13T09:22:30.609 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:22:30.610 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T09:22:30.610 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:22:30.676 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rbd 2024-09-13T09:22:30.838 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-nbd-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:31.140 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-nbd-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:31.140 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:31.140 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-13T09:22:31.140 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:22:31.140 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:31.140 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:22:31.240 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:37 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:22:31.330 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rbd-2:18.2.4-820.g48a7de7e.el9.x86_64 is already installed. 2024-09-13T09:22:31.373 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:22:31.374 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T09:22:31.375 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:22:31.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-mirror 2024-09-13T09:22:31.446 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-fuse 2024-09-13T09:22:31.962 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 09:20:46 AM UTC. 2024-09-13T09:22:32.008 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:38 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:22:32.055 INFO:teuthology.orchestra.run.smithi190.stdout:Package rbd-mirror-2:18.2.4-820.g48a7de7e.el9.x86_64 is already installed. 2024-09-13T09:22:32.100 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-13T09:22:32.101 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-13T09:22:32.102 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-13T09:22:32.141 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:22:32.142 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:32.142 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-09-13T09:22:32.142 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:32.142 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T09:22:32.142 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 87 k 2024-09-13T09:22:32.142 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:32.143 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T09:22:32.143 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:32.143 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T09:22:32.143 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:32.143 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 87 k 2024-09-13T09:22:32.143 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 227 k 2024-09-13T09:22:32.143 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T09:22:32.172 DEBUG:teuthology.parallel:result is None 2024-09-13T09:22:32.330 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-fuse-18.2.4-820.g48a7de7e.el9.x86_64.rpm 465 kB/s | 87 kB 00:00 2024-09-13T09:22:32.331 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:22:32.331 INFO:teuthology.orchestra.run.smithi122.stdout:Total 461 kB/s | 87 kB 00:00 2024-09-13T09:22:32.331 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T09:22:32.338 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T09:22:32.338 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T09:22:32.401 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T09:22:32.402 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T09:22:32.566 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T09:22:32.677 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-fuse-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:33.047 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-fuse-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:33.311 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-fuse-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:33.312 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:33.312 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T09:22:33.312 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:22:33.312 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:33.312 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:22:33.598 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-mirror 2024-09-13T09:22:34.174 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:40 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:22:34.311 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:22:34.312 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:34.312 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-09-13T09:22:34.312 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:34.312 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T09:22:34.312 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 3.0 M 2024-09-13T09:22:34.312 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:34.312 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T09:22:34.312 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:34.312 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T09:22:34.312 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:34.313 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 3.0 M 2024-09-13T09:22:34.313 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 12 M 2024-09-13T09:22:34.313 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T09:22:34.617 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-mirror-18.2.4-820.g48a7de7e.el9.x86_64.rpm 9.8 MB/s | 3.0 MB 00:00 2024-09-13T09:22:34.618 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:22:34.618 INFO:teuthology.orchestra.run.smithi122.stdout:Total 9.8 MB/s | 3.0 MB 00:00 2024-09-13T09:22:34.618 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T09:22:34.627 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T09:22:34.628 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T09:22:34.708 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T09:22:34.708 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T09:22:35.125 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T09:22:35.185 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-mirror-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:35.209 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-mirror-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:35.209 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T09:22:35.209 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T09:22:35.209 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T09:22:35.209 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T09:22:35.209 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:36.317 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-mirror-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:36.317 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:36.318 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T09:22:36.318 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:22:36.318 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:36.318 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:22:36.574 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-nbd 2024-09-13T09:22:37.162 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:43 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:22:37.297 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:22:37.298 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:37.298 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-09-13T09:22:37.298 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:37.298 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T09:22:37.298 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd x86_64 2:18.2.4-820.g48a7de7e.el9 ceph 172 k 2024-09-13T09:22:37.298 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:37.298 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T09:22:37.299 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T09:22:37.299 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T09:22:37.299 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:37.299 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 172 k 2024-09-13T09:22:37.299 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 494 k 2024-09-13T09:22:37.299 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T09:22:37.497 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-nbd-18.2.4-820.g48a7de7e.el9.x86_64.rpm 870 kB/s | 172 kB 00:00 2024-09-13T09:22:37.497 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T09:22:37.497 INFO:teuthology.orchestra.run.smithi122.stdout:Total 863 kB/s | 172 kB 00:00 2024-09-13T09:22:37.498 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T09:22:37.505 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T09:22:37.505 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T09:22:37.570 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T09:22:37.570 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T09:22:37.724 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T09:22:37.829 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-nbd-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:38.273 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-nbd-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:38.532 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-nbd-2:18.2.4-820.g48a7de7e.el9.x86_64 1/1 2024-09-13T09:22:38.532 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:38.533 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T09:22:38.533 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd-2:18.2.4-820.g48a7de7e.el9.x86_64 2024-09-13T09:22:38.533 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:38.533 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:22:38.810 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-mirror 2024-09-13T09:22:39.390 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 09:20:54 AM UTC. 2024-09-13T09:22:39.481 INFO:teuthology.orchestra.run.smithi122.stdout:Package rbd-mirror-2:18.2.4-820.g48a7de7e.el9.x86_64 is already installed. 2024-09-13T09:22:39.526 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T09:22:39.527 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T09:22:39.527 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T09:22:39.596 DEBUG:teuthology.parallel:result is None 2024-09-13T09:22:39.597 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=48a7de7e6e1701a88fd571c7e34908ac4b8ed412 2024-09-13T09:22:39.737 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T09:22:39.758 INFO:teuthology.orchestra.run.smithi122.stdout:18.2.4-820.g48a7de7e.el9 2024-09-13T09:22:39.759 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-820.g48a7de7e.el9 2024-09-13T09:22:39.759 INFO:teuthology.task.install:The correct ceph version 18.2.4-820.g48a7de7e is installed. 2024-09-13T09:22:39.761 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=48a7de7e6e1701a88fd571c7e34908ac4b8ed412 2024-09-13T09:22:39.901 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T09:22:39.926 INFO:teuthology.orchestra.run.smithi190.stdout:18.2.4-820.g48a7de7e.el9 2024-09-13T09:22:39.926 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-820.g48a7de7e.el9 2024-09-13T09:22:39.926 INFO:teuthology.task.install:The correct ceph version 18.2.4-820.g48a7de7e is installed. 2024-09-13T09:22:39.928 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T09:22:39.929 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:22:39.929 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T09:22:39.954 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:22:39.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T09:22:39.994 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T09:22:39.995 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:22:39.995 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T09:22:40.019 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T09:22:40.083 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:22:40.083 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T09:22:40.111 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T09:22:40.178 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T09:22:40.178 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:22:40.178 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T09:22:40.203 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T09:22:40.268 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:22:40.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T09:22:40.295 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T09:22:40.362 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T09:22:40.362 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:22:40.362 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T09:22:40.387 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T09:22:40.452 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:22:40.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T09:22:40.478 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T09:22:40.544 INFO:teuthology.run_tasks:Running task ceph... 2024-09-13T09:22:40.619 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-13T09:22:40.619 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /var/log/ceph 2024-09-13T09:22:40.622 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /var/log/ceph 2024-09-13T09:22:40.647 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-13T09:22:40.647 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T09:22:40.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T09:22:40.712 INFO:tasks.ceph:Creating extra log directories... 2024-09-13T09:22:40.712 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T09:22:40.750 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T09:22:40.788 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-09-13T09:22:40.789 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-13T09:22:40.789 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7901034', 'branch': 'reef', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7901034', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': '7a853b1706d076d58e0da7a5d3b9b9eb36276880'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 248, 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412', 'sleep_before_teardown': 0, 'subset': '116/128', 'suite': 'rbd', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_7a853b1706d076d58e0da7a5d3b9b9eb36276880/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '7a853b1706d076d58e0da7a5d3b9b9eb36276880', 'targets': {'smithi122.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBPLTJok4PZJYDU2rwScvJ7rB23VRq1g0ehgis1fGIuL1AVwW47oni1FTJmM56obzAHWh8Hf7+Ni6NVwEMRUVuA=', 'smithi190.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD/2hn0Bf/KzJKwZT28wcQLhGlXAxiKjAtWMHBFSyYnSZxSL3Z81UbS2xmNOY2cVn5onuYGXxBkdym2Qn+M0/v8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_22:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-13T09:22:40.789 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-13T09:22:40.815 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-13T09:22:40.833 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T09:22:40.870 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T09:22:40.897 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:22:40.897 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T09:22:40.948 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T09:22:40.949 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_1 2024-09-13T09:22:41.003 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T09:22:41.003 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T09:22:41.003 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-13T09:22:41.003 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T09:22:41.003 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T09:22:41.003 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-09-13 09:22:35.467939533 +0000 2024-09-13T09:22:41.003 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-09-13 09:20:09.985702286 +0000 2024-09-13T09:22:41.003 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-09-13 09:20:09.985702286 +0000 2024-09-13T09:22:41.003 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-09-13 09:20:09.985702286 +0000 2024-09-13T09:22:41.004 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T09:22:41.066 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-09-13T09:22:41.066 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-09-13T09:22:41.066 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000242633 s, 2.1 MB/s 2024-09-13T09:22:41.067 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T09:22:41.123 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_2 2024-09-13T09:22:41.178 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T09:22:41.178 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T09:22:41.178 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-13T09:22:41.178 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T09:22:41.178 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T09:22:41.178 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-09-13 09:22:35.468939509 +0000 2024-09-13T09:22:41.178 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-09-13 09:20:10.302694623 +0000 2024-09-13T09:22:41.178 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-09-13 09:20:10.302694623 +0000 2024-09-13T09:22:41.178 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-09-13 09:20:10.302694623 +0000 2024-09-13T09:22:41.179 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T09:22:41.241 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-09-13T09:22:41.241 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-09-13T09:22:41.241 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.00019898 s, 2.6 MB/s 2024-09-13T09:22:41.242 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T09:22:41.298 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_3 2024-09-13T09:22:41.353 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T09:22:41.353 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T09:22:41.353 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-13T09:22:41.353 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T09:22:41.353 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T09:22:41.353 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-09-13 09:22:35.468939509 +0000 2024-09-13T09:22:41.353 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-09-13 09:20:10.636686549 +0000 2024-09-13T09:22:41.353 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-09-13 09:20:10.636686549 +0000 2024-09-13T09:22:41.353 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-09-13 09:20:10.636686549 +0000 2024-09-13T09:22:41.354 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T09:22:41.416 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-09-13T09:22:41.416 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-09-13T09:22:41.416 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000206928 s, 2.5 MB/s 2024-09-13T09:22:41.417 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T09:22:41.473 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_4 2024-09-13T09:22:41.528 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T09:22:41.529 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T09:22:41.529 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-13T09:22:41.529 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T09:22:41.529 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T09:22:41.529 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-09-13 09:22:35.469939484 +0000 2024-09-13T09:22:41.529 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-09-13 09:20:10.970678476 +0000 2024-09-13T09:22:41.529 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-09-13 09:20:10.970678476 +0000 2024-09-13T09:22:41.529 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-09-13 09:20:10.970678476 +0000 2024-09-13T09:22:41.529 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T09:22:41.592 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-09-13T09:22:41.592 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-09-13T09:22:41.592 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000189677 s, 2.7 MB/s 2024-09-13T09:22:41.593 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T09:22:41.649 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T09:22:41.649 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi122.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-13T09:22:41.649 INFO:tasks.ceph:Generating config... 2024-09-13T09:22:41.650 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-13T09:22:41.650 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-13T09:22:41.650 INFO:tasks.ceph:[client] log to stderr = False 2024-09-13T09:22:41.650 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-13T09:22:41.650 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-09-13T09:22:41.651 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:22:41.651 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:22:41.651 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:22:41.651 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:22:41.651 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:22:41.651 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:22:41.651 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:22:41.651 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:22:41.651 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-13T09:22:41.651 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-13T09:22:41.651 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-13T09:22:41.651 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-13T09:22:41.651 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-13T09:22:41.651 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-13T09:22:41.651 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-13T09:22:41.652 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-13T09:22:41.652 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-13T09:22:41.652 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-13T09:22:41.652 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-13T09:22:41.652 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-13T09:22:41.652 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-13T09:22:41.652 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-13T09:22:41.652 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-13T09:22:41.652 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-13T09:22:41.652 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-13T09:22:41.652 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-13T09:22:41.652 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-13T09:22:41.652 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-13T09:22:41.652 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-13T09:22:41.652 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-13T09:22:41.653 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-09-13T09:22:41.653 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-09-13T09:22:41.751 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster1.keyring 2024-09-13T09:22:41.753 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-09-13T09:22:41.795 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-13T09:22:41.859 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.122')] 2024-09-13T09:22:41.860 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.122', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-13T09:22:41.861 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:22:41.861 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T09:22:41.915 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.122 --print /home/ubuntu/cephtest/cluster1.monmap 2024-09-13T09:22:41.995 INFO:teuthology.orchestra.run.smithi122.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-09-13T09:22:41.995 INFO:teuthology.orchestra.run.smithi122.stdout:monmaptool: generated fsid b196829d-0abf-49d7-afed-e85b83e5f85c 2024-09-13T09:22:41.995 INFO:teuthology.orchestra.run.smithi122.stdout:setting min_mon_release = pacific 2024-09-13T09:22:41.995 INFO:teuthology.orchestra.run.smithi122.stdout:epoch 0 2024-09-13T09:22:41.996 INFO:teuthology.orchestra.run.smithi122.stdout:fsid b196829d-0abf-49d7-afed-e85b83e5f85c 2024-09-13T09:22:41.996 INFO:teuthology.orchestra.run.smithi122.stdout:last_changed 2024-09-13T09:22:41.994808+0000 2024-09-13T09:22:41.996 INFO:teuthology.orchestra.run.smithi122.stdout:created 2024-09-13T09:22:41.994808+0000 2024-09-13T09:22:41.996 INFO:teuthology.orchestra.run.smithi122.stdout:min_mon_release 16 (pacific) 2024-09-13T09:22:41.996 INFO:teuthology.orchestra.run.smithi122.stdout:election_strategy: 1 2024-09-13T09:22:41.996 INFO:teuthology.orchestra.run.smithi122.stdout:0: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.a 2024-09-13T09:22:41.996 INFO:teuthology.orchestra.run.smithi122.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-09-13T09:22:41.997 DEBUG:teuthology.orchestra.run.smithi122:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T09:22:42.052 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID b196829d-0abf-49d7-afed-e85b83e5f85c... 2024-09-13T09:22:42.053 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-13T09:22:42.094 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-13T09:22:42.135 INFO:teuthology.orchestra.run.smithi122.stdout:[global] 2024-09-13T09:22:42.135 INFO:teuthology.orchestra.run.smithi122.stdout: chdir = "" 2024-09-13T09:22:42.135 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T09:22:42.135 INFO:teuthology.orchestra.run.smithi122.stdout: auth supported = cephx 2024-09-13T09:22:42.135 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.135 INFO:teuthology.orchestra.run.smithi122.stdout: filestore xattr use omap = true 2024-09-13T09:22:42.136 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.136 INFO:teuthology.orchestra.run.smithi122.stdout: mon clock drift allowed = 1.000 2024-09-13T09:22:42.136 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.136 INFO:teuthology.orchestra.run.smithi122.stdout: osd crush chooseleaf type = 0 2024-09-13T09:22:42.136 INFO:teuthology.orchestra.run.smithi122.stdout: auth debug = true 2024-09-13T09:22:42.136 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.136 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on old message = true 2024-09-13T09:22:42.136 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on bug = true 2024-09-13T09:22:42.136 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.136 INFO:teuthology.orchestra.run.smithi122.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T09:22:42.136 INFO:teuthology.orchestra.run.smithi122.stdout: mon pg warn max object skew = 0 2024-09-13T09:22:42.136 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.136 INFO:teuthology.orchestra.run.smithi122.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T09:22:42.137 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T09:22:42.137 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.137 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default size = 2 2024-09-13T09:22:42.137 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.137 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow primary affinity = true 2024-09-13T09:22:42.137 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow pg remap = true 2024-09-13T09:22:42.137 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on legacy crush tunables = false 2024-09-13T09:22:42.137 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on crush straw calc version zero = false 2024-09-13T09:22:42.137 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on no sortbitwise = false 2024-09-13T09:22:42.137 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on osd down out interval zero = false 2024-09-13T09:22:42.137 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on too few osds = false 2024-09-13T09:22:42.137 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T09:22:42.137 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T09:22:42.138 INFO:teuthology.orchestra.run.smithi122.stdout: mon_allow_pool_size_one = true 2024-09-13T09:22:42.138 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.138 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T09:22:42.138 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.138 INFO:teuthology.orchestra.run.smithi122.stdout: osd default data pool replay window = 5 2024-09-13T09:22:42.138 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.138 INFO:teuthology.orchestra.run.smithi122.stdout: mon allow pool delete = true 2024-09-13T09:22:42.138 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.138 INFO:teuthology.orchestra.run.smithi122.stdout: mon cluster log file level = debug 2024-09-13T09:22:42.138 INFO:teuthology.orchestra.run.smithi122.stdout: debug asserts on shutdown = true 2024-09-13T09:22:42.138 INFO:teuthology.orchestra.run.smithi122.stdout: mon health detail to clog = false 2024-09-13T09:22:42.138 INFO:teuthology.orchestra.run.smithi122.stdout: mon host = 172.21.15.122 2024-09-13T09:22:42.138 INFO:teuthology.orchestra.run.smithi122.stdout: mon client directed command retry = 5 2024-09-13T09:22:42.138 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on pool no app = False 2024-09-13T09:22:42.139 INFO:teuthology.orchestra.run.smithi122.stdout: ms inject socket failures = 5000 2024-09-13T09:22:42.139 INFO:teuthology.orchestra.run.smithi122.stdout: fsid = b196829d-0abf-49d7-afed-e85b83e5f85c 2024-09-13T09:22:42.139 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.139 INFO:teuthology.orchestra.run.smithi122.stdout:[osd] 2024-09-13T09:22:42.139 INFO:teuthology.orchestra.run.smithi122.stdout: osd journal size = 100 2024-09-13T09:22:42.139 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.139 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub load threshold = 5.0 2024-09-13T09:22:42.139 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub max interval = 600 2024-09-13T09:22:42.139 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock profile = high_recovery_ops 2024-09-13T09:22:42.139 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.139 INFO:teuthology.orchestra.run.smithi122.stdout: osd recover clone overlap = true 2024-09-13T09:22:42.139 INFO:teuthology.orchestra.run.smithi122.stdout: osd recovery max chunk = 1048576 2024-09-13T09:22:42.139 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.140 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug shutdown = true 2024-09-13T09:22:42.140 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug op order = true 2024-09-13T09:22:42.140 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify stray on activate = true 2024-09-13T09:22:42.140 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.140 INFO:teuthology.orchestra.run.smithi122.stdout: osd open classes on start = true 2024-09-13T09:22:42.140 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug pg log writeout = true 2024-09-13T09:22:42.140 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.140 INFO:teuthology.orchestra.run.smithi122.stdout: osd deep scrub update digest min age = 30 2024-09-13T09:22:42.140 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.140 INFO:teuthology.orchestra.run.smithi122.stdout: osd map max advance = 10 2024-09-13T09:22:42.140 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.140 INFO:teuthology.orchestra.run.smithi122.stdout: journal zero on create = true 2024-09-13T09:22:42.140 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.140 INFO:teuthology.orchestra.run.smithi122.stdout: filestore ondisk finisher threads = 3 2024-09-13T09:22:42.140 INFO:teuthology.orchestra.run.smithi122.stdout: filestore apply finisher threads = 3 2024-09-13T09:22:42.141 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.141 INFO:teuthology.orchestra.run.smithi122.stdout: bdev debug aio = true 2024-09-13T09:22:42.141 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug misdirected ops = true 2024-09-13T09:22:42.141 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore block size = 96636764160 2024-09-13T09:22:42.141 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore compression algorithm = snappy 2024-09-13T09:22:42.141 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore compression mode = aggressive 2024-09-13T09:22:42.141 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore fsck on mount = True 2024-09-13T09:22:42.141 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluefs = 1/20 2024-09-13T09:22:42.141 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluestore = 1/20 2024-09-13T09:22:42.141 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-09-13T09:22:42.141 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 20 2024-09-13T09:22:42.141 INFO:teuthology.orchestra.run.smithi122.stdout: debug rocksdb = 4/10 2024-09-13T09:22:42.141 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T09:22:42.142 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd full ratio = 0.9 2024-09-13T09:22:42.142 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd nearfull ratio = 0.8 2024-09-13T09:22:42.142 INFO:teuthology.orchestra.run.smithi122.stdout: osd failsafe full ratio = 0.95 2024-09-13T09:22:42.142 INFO:teuthology.orchestra.run.smithi122.stdout: osd objectstore = bluestore 2024-09-13T09:22:42.142 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.142 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr] 2024-09-13T09:22:42.142 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-09-13T09:22:42.142 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 20 2024-09-13T09:22:42.142 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-09-13T09:22:42.142 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-09-13T09:22:42.142 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-09-13T09:22:42.142 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-09-13T09:22:42.142 INFO:teuthology.orchestra.run.smithi122.stdout: mgr/telemetry/nag = false 2024-09-13T09:22:42.143 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.143 INFO:teuthology.orchestra.run.smithi122.stdout:[mon] 2024-09-13T09:22:42.143 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-09-13T09:22:42.143 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-09-13T09:22:42.143 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 20 2024-09-13T09:22:42.143 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-09-13T09:22:42.143 INFO:teuthology.orchestra.run.smithi122.stdout: mon data avail warn = 5 2024-09-13T09:22:42.143 INFO:teuthology.orchestra.run.smithi122.stdout: mon mgr mkfs grace = 240 2024-09-13T09:22:42.143 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-09-13T09:22:42.143 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd reporter subtree level = osd 2024-09-13T09:22:42.143 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd prime pg temp = true 2024-09-13T09:22:42.143 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-09-13T09:22:42.143 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.143 INFO:teuthology.orchestra.run.smithi122.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T09:22:42.144 INFO:teuthology.orchestra.run.smithi122.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T09:22:42.144 INFO:teuthology.orchestra.run.smithi122.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T09:22:42.144 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.144 INFO:teuthology.orchestra.run.smithi122.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T09:22:42.144 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T09:22:42.144 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T09:22:42.144 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.144 INFO:teuthology.orchestra.run.smithi122.stdout: # 1m isn't quite enough 2024-09-13T09:22:42.144 INFO:teuthology.orchestra.run.smithi122.stdout: mon_down_mkfs_grace = 2m 2024-09-13T09:22:42.144 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.144 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_filestore_osds = false 2024-09-13T09:22:42.144 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:42.144 INFO:teuthology.orchestra.run.smithi122.stdout:[client] 2024-09-13T09:22:42.144 INFO:teuthology.orchestra.run.smithi122.stdout: rgw cache enabled = true 2024-09-13T09:22:42.144 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable ops log = true 2024-09-13T09:22:42.145 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable usage log = true 2024-09-13T09:22:42.145 INFO:teuthology.orchestra.run.smithi122.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T09:22:42.145 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T09:22:42.145 INFO:teuthology.orchestra.run.smithi122.stdout: debug rbd = 20 2024-09-13T09:22:42.145 INFO:teuthology.orchestra.run.smithi122.stdout: debug rbd_mirror = 15 2024-09-13T09:22:42.145 INFO:teuthology.orchestra.run.smithi122.stdout: log to stderr = False 2024-09-13T09:22:42.145 INFO:teuthology.orchestra.run.smithi122.stdout: rbd default features = 125 2024-09-13T09:22:42.145 INFO:teuthology.orchestra.run.smithi122.stdout: rbd mirror image policy type = simple 2024-09-13T09:22:42.145 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.a] 2024-09-13T09:22:42.145 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.0] 2024-09-13T09:22:42.145 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:22:42.145 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:22:42.145 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.1] 2024-09-13T09:22:42.145 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:22:42.146 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:22:42.146 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.2] 2024-09-13T09:22:42.146 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:22:42.146 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:22:42.146 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.3] 2024-09-13T09:22:42.146 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:22:42.146 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:22:42.146 INFO:teuthology.orchestra.run.smithi190.stdout:[global] 2024-09-13T09:22:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: chdir = "" 2024-09-13T09:22:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T09:22:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: auth supported = cephx 2024-09-13T09:22:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: filestore xattr use omap = true 2024-09-13T09:22:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: mon clock drift allowed = 1.000 2024-09-13T09:22:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: osd crush chooseleaf type = 0 2024-09-13T09:22:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: auth debug = true 2024-09-13T09:22:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on old message = true 2024-09-13T09:22:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on bug = true 2024-09-13T09:22:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.148 INFO:teuthology.orchestra.run.smithi190.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T09:22:42.148 INFO:teuthology.orchestra.run.smithi190.stdout: mon pg warn max object skew = 0 2024-09-13T09:22:42.148 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.148 INFO:teuthology.orchestra.run.smithi190.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T09:22:42.148 INFO:teuthology.orchestra.run.smithi190.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T09:22:42.148 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.148 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default size = 2 2024-09-13T09:22:42.148 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.148 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow primary affinity = true 2024-09-13T09:22:42.148 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow pg remap = true 2024-09-13T09:22:42.148 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on legacy crush tunables = false 2024-09-13T09:22:42.148 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on crush straw calc version zero = false 2024-09-13T09:22:42.148 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on no sortbitwise = false 2024-09-13T09:22:42.148 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on osd down out interval zero = false 2024-09-13T09:22:42.149 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on too few osds = false 2024-09-13T09:22:42.149 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T09:22:42.149 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T09:22:42.149 INFO:teuthology.orchestra.run.smithi190.stdout: mon_allow_pool_size_one = true 2024-09-13T09:22:42.149 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.149 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T09:22:42.149 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.149 INFO:teuthology.orchestra.run.smithi190.stdout: osd default data pool replay window = 5 2024-09-13T09:22:42.149 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.149 INFO:teuthology.orchestra.run.smithi190.stdout: mon allow pool delete = true 2024-09-13T09:22:42.149 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.149 INFO:teuthology.orchestra.run.smithi190.stdout: mon cluster log file level = debug 2024-09-13T09:22:42.149 INFO:teuthology.orchestra.run.smithi190.stdout: debug asserts on shutdown = true 2024-09-13T09:22:42.149 INFO:teuthology.orchestra.run.smithi190.stdout: mon health detail to clog = false 2024-09-13T09:22:42.149 INFO:teuthology.orchestra.run.smithi190.stdout: mon host = 172.21.15.122 2024-09-13T09:22:42.150 INFO:teuthology.orchestra.run.smithi190.stdout: mon client directed command retry = 5 2024-09-13T09:22:42.150 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on pool no app = False 2024-09-13T09:22:42.150 INFO:teuthology.orchestra.run.smithi190.stdout: ms inject socket failures = 5000 2024-09-13T09:22:42.150 INFO:teuthology.orchestra.run.smithi190.stdout: fsid = b196829d-0abf-49d7-afed-e85b83e5f85c 2024-09-13T09:22:42.150 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.150 INFO:teuthology.orchestra.run.smithi190.stdout:[osd] 2024-09-13T09:22:42.150 INFO:teuthology.orchestra.run.smithi190.stdout: osd journal size = 100 2024-09-13T09:22:42.150 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.150 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub load threshold = 5.0 2024-09-13T09:22:42.150 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub max interval = 600 2024-09-13T09:22:42.150 INFO:teuthology.orchestra.run.smithi190.stdout: osd mclock profile = high_recovery_ops 2024-09-13T09:22:42.150 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.150 INFO:teuthology.orchestra.run.smithi190.stdout: osd recover clone overlap = true 2024-09-13T09:22:42.150 INFO:teuthology.orchestra.run.smithi190.stdout: osd recovery max chunk = 1048576 2024-09-13T09:22:42.150 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.151 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug shutdown = true 2024-09-13T09:22:42.151 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug op order = true 2024-09-13T09:22:42.151 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify stray on activate = true 2024-09-13T09:22:42.151 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.151 INFO:teuthology.orchestra.run.smithi190.stdout: osd open classes on start = true 2024-09-13T09:22:42.151 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug pg log writeout = true 2024-09-13T09:22:42.151 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.151 INFO:teuthology.orchestra.run.smithi190.stdout: osd deep scrub update digest min age = 30 2024-09-13T09:22:42.151 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.151 INFO:teuthology.orchestra.run.smithi190.stdout: osd map max advance = 10 2024-09-13T09:22:42.152 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.152 INFO:teuthology.orchestra.run.smithi190.stdout: journal zero on create = true 2024-09-13T09:22:42.152 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.152 INFO:teuthology.orchestra.run.smithi190.stdout: filestore ondisk finisher threads = 3 2024-09-13T09:22:42.152 INFO:teuthology.orchestra.run.smithi190.stdout: filestore apply finisher threads = 3 2024-09-13T09:22:42.152 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.152 INFO:teuthology.orchestra.run.smithi190.stdout: bdev debug aio = true 2024-09-13T09:22:42.152 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug misdirected ops = true 2024-09-13T09:22:42.152 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore block size = 96636764160 2024-09-13T09:22:42.152 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore compression algorithm = snappy 2024-09-13T09:22:42.152 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore compression mode = aggressive 2024-09-13T09:22:42.152 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore fsck on mount = True 2024-09-13T09:22:42.152 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluefs = 1/20 2024-09-13T09:22:42.152 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluestore = 1/20 2024-09-13T09:22:42.153 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-13T09:22:42.153 INFO:teuthology.orchestra.run.smithi190.stdout: debug osd = 20 2024-09-13T09:22:42.153 INFO:teuthology.orchestra.run.smithi190.stdout: debug rocksdb = 4/10 2024-09-13T09:22:42.153 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T09:22:42.153 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd full ratio = 0.9 2024-09-13T09:22:42.153 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd nearfull ratio = 0.8 2024-09-13T09:22:42.153 INFO:teuthology.orchestra.run.smithi190.stdout: osd failsafe full ratio = 0.95 2024-09-13T09:22:42.153 INFO:teuthology.orchestra.run.smithi190.stdout: osd objectstore = bluestore 2024-09-13T09:22:42.153 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.153 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr] 2024-09-13T09:22:42.153 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-13T09:22:42.153 INFO:teuthology.orchestra.run.smithi190.stdout: debug mgr = 20 2024-09-13T09:22:42.153 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2024-09-13T09:22:42.153 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2024-09-13T09:22:42.153 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2024-09-13T09:22:42.154 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2024-09-13T09:22:42.154 INFO:teuthology.orchestra.run.smithi190.stdout: mgr/telemetry/nag = false 2024-09-13T09:22:42.154 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.154 INFO:teuthology.orchestra.run.smithi190.stdout:[mon] 2024-09-13T09:22:42.154 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-13T09:22:42.154 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2024-09-13T09:22:42.154 INFO:teuthology.orchestra.run.smithi190.stdout: debug paxos = 20 2024-09-13T09:22:42.154 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2024-09-13T09:22:42.154 INFO:teuthology.orchestra.run.smithi190.stdout: mon data avail warn = 5 2024-09-13T09:22:42.154 INFO:teuthology.orchestra.run.smithi190.stdout: mon mgr mkfs grace = 240 2024-09-13T09:22:42.154 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2024-09-13T09:22:42.154 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd reporter subtree level = osd 2024-09-13T09:22:42.155 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd prime pg temp = true 2024-09-13T09:22:42.155 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2024-09-13T09:22:42.155 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.155 INFO:teuthology.orchestra.run.smithi190.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T09:22:42.155 INFO:teuthology.orchestra.run.smithi190.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T09:22:42.155 INFO:teuthology.orchestra.run.smithi190.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T09:22:42.155 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.155 INFO:teuthology.orchestra.run.smithi190.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T09:22:42.155 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T09:22:42.155 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T09:22:42.155 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.155 INFO:teuthology.orchestra.run.smithi190.stdout: # 1m isn't quite enough 2024-09-13T09:22:42.155 INFO:teuthology.orchestra.run.smithi190.stdout: mon_down_mkfs_grace = 2m 2024-09-13T09:22:42.155 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.155 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_filestore_osds = false 2024-09-13T09:22:42.156 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:22:42.156 INFO:teuthology.orchestra.run.smithi190.stdout:[client] 2024-09-13T09:22:42.156 INFO:teuthology.orchestra.run.smithi190.stdout: rgw cache enabled = true 2024-09-13T09:22:42.156 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable ops log = true 2024-09-13T09:22:42.156 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable usage log = true 2024-09-13T09:22:42.156 INFO:teuthology.orchestra.run.smithi190.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T09:22:42.156 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T09:22:42.156 INFO:teuthology.orchestra.run.smithi190.stdout: debug rbd = 20 2024-09-13T09:22:42.156 INFO:teuthology.orchestra.run.smithi190.stdout: debug rbd_mirror = 15 2024-09-13T09:22:42.156 INFO:teuthology.orchestra.run.smithi190.stdout: log to stderr = False 2024-09-13T09:22:42.156 INFO:teuthology.orchestra.run.smithi190.stdout: rbd default features = 125 2024-09-13T09:22:42.156 INFO:teuthology.orchestra.run.smithi190.stdout: rbd mirror image policy type = simple 2024-09-13T09:22:42.156 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.a] 2024-09-13T09:22:42.156 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.0] 2024-09-13T09:22:42.156 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:22:42.157 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:22:42.157 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.1] 2024-09-13T09:22:42.157 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:22:42.157 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:22:42.157 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.2] 2024-09-13T09:22:42.157 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:22:42.157 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:22:42.157 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.3] 2024-09-13T09:22:42.157 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:22:42.157 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:22:42.158 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-09-13T09:22:42.159 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-09-13T09:22:42.251 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-13T09:22:42.251 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:22:42.251 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-09-13T09:22:42.267 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:22:42.267 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-09-13T09:22:42.322 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T09:22:42.322 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:22:42.322 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-13T09:22:42.322 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-13T09:22:42.395 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:22:42.396 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-13T09:22:42.451 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi190.front.sepia.ceph.com 2024-09-13T09:22:42.451 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:22:42.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-13T09:22:42.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-13T09:22:42.493 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:22:42.493 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-13T09:22:42.551 INFO:tasks.ceph:Setting up mon nodes... 2024-09-13T09:22:42.551 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-13T09:22:42.551 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-13T09:22:42.600 INFO:teuthology.orchestra.run.smithi122.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-13T09:22:42.603 INFO:tasks.ceph:Setting up mds nodes... 2024-09-13T09:22:42.603 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-13T09:22:42.604 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-09-13T09:22:42.644 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-09-13T09:22:42.656 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-09-13T09:22:42.711 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-09-13T09:22:42.724 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-13T09:22:42.766 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-13T09:22:42.779 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-13T09:22:42.861 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-13T09:22:42.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-13T09:22:42.953 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-13T09:22:42.966 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-13T09:22:43.047 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-13T09:22:43.059 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-13T09:22:43.140 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-13T09:22:43.153 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-13T09:22:43.234 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-13T09:22:43.246 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-13T09:22:43.327 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-13T09:22:43.339 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-13T09:22:43.339 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi122.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-13T09:22:43.339 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-09-13T09:22:43.369 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T09:22:43.369 INFO:tasks.ceph:role: cluster1.osd.0 2024-09-13T09:22:43.369 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T09:22:43.369 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T09:22:43.445 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T09:22:43.445 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T09:22:43.445 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T09:22:43.445 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T09:22:43.445 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T09:22:43.445 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-09-13T09:22:43.445 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T09:22:43.445 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T09:22:43.445 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T09:22:43.445 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T09:22:43.949 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-09-13T09:22:43.951 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-09-13T09:22:43.951 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-09-13T09:22:44.067 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-09-13T09:22:44.103 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-09-13T09:22:44.174 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T09:22:44.174 INFO:tasks.ceph:role: cluster1.osd.1 2024-09-13T09:22:44.174 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T09:22:44.174 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T09:22:44.252 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T09:22:44.252 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T09:22:44.253 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T09:22:44.253 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T09:22:44.253 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T09:22:44.253 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-09-13T09:22:44.253 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T09:22:44.253 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T09:22:44.253 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T09:22:44.253 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T09:22:44.708 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-09-13T09:22:44.710 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-09-13T09:22:44.711 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-09-13T09:22:44.802 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-09-13T09:22:44.833 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-09-13T09:22:44.899 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T09:22:44.899 INFO:tasks.ceph:role: cluster1.osd.2 2024-09-13T09:22:44.899 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T09:22:44.899 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T09:22:44.972 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T09:22:44.972 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T09:22:44.972 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T09:22:44.973 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T09:22:44.973 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T09:22:44.973 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-09-13T09:22:44.973 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T09:22:44.973 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T09:22:44.973 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T09:22:44.973 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T09:22:45.459 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-09-13T09:22:45.461 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-09-13T09:22:45.461 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-09-13T09:22:45.583 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-09-13T09:22:45.617 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-13T09:22:45.721 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T09:22:45.720+0000 7f76fef30640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-09-13T09:22:45.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T09:22:45.725+0000 7f76fef30640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-09-13T09:22:45.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T09:22:45.725+0000 7f76fef30640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-09-13T09:22:45.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T09:22:45.725+0000 7f76fef30640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-09-13T09:22:45.727 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T09:22:45.725+0000 7f76fef30640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-09-13T09:22:45.731 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T09:22:45.730+0000 7f76fef30640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-09-13T09:22:48.578 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-09-13T09:22:48.609 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-13T09:22:48.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T09:22:48.701+0000 7f9758e11640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-09-13T09:22:48.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T09:22:48.702+0000 7f9758e11640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-09-13T09:22:48.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T09:22:48.702+0000 7f9758e11640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-09-13T09:22:48.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T09:22:48.702+0000 7f9758e11640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-09-13T09:22:48.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T09:22:48.702+0000 7f9758e11640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-09-13T09:22:48.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T09:22:48.702+0000 7f9758e11640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-09-13T09:22:51.555 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-09-13T09:22:51.588 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-13T09:22:51.679 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T09:22:51.678+0000 7f58f6f1e640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-09-13T09:22:51.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T09:22:51.679+0000 7f58f6f1e640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-09-13T09:22:51.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T09:22:51.679+0000 7f58f6f1e640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-09-13T09:22:51.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T09:22:51.679+0000 7f58f6f1e640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-09-13T09:22:51.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T09:22:51.679+0000 7f58f6f1e640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-09-13T09:22:51.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T09:22:51.679+0000 7f58f6f1e640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-09-13T09:22:54.539 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-09-13T09:22:54.565 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-13T09:22:54.565 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:22:54.565 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-09-13T09:22:54.632 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:22:54.632 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-09-13T09:22:54.697 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:22:54.697 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-09-13T09:22:54.762 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:22:54.762 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-09-13T09:22:54.827 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:22:54.827 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-09-13T09:22:54.882 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:22:54.882 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-09-13T09:22:54.902 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:22:54.902 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-09-13T09:22:54.961 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:22:54.961 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-09-13T09:22:55.019 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:22:55.019 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-09-13T09:22:55.077 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:22:55.078 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-09-13T09:22:55.135 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:22:55.135 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-09-13T09:22:55.193 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:22:55.193 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-09-13T09:22:55.250 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:22:55.250 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-09-13T09:22:55.308 INFO:tasks.ceph:Adding keys to all mons... 2024-09-13T09:22:55.308 DEBUG:teuthology.orchestra.run.smithi122:> sudo tee -a /etc/ceph/cluster1.keyring 2024-09-13T09:22:55.332 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr.x] 2024-09-13T09:22:55.332 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBiBORmNnnNIxAA6MWuxIqbmy89bkabUDTKag== 2024-09-13T09:22:55.332 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.0] 2024-09-13T09:22:55.332 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBlBORmZIP9KhAAr6d0nOuKLbgHC+E5J8qOdA== 2024-09-13T09:22:55.332 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.1] 2024-09-13T09:22:55.332 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBoBORmbiLjKRAAffdAZ6dvwDJ2wFMh6ar7aw== 2024-09-13T09:22:55.332 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.2] 2024-09-13T09:22:55.332 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBrBORm/VWEKBAAhY4iS8s/jBRE9mAIgR7RaQ== 2024-09-13T09:22:55.332 INFO:teuthology.orchestra.run.smithi122.stdout:[client.0] 2024-09-13T09:22:55.333 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBiBORmkP9gJhAAQysNPag5aAwWRl8WhUUBnA== 2024-09-13T09:22:55.333 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror] 2024-09-13T09:22:55.333 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBiBORm2ghlKhAApMmJgkANmX2XbSGzsDvy9A== 2024-09-13T09:22:55.333 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.0] 2024-09-13T09:22:55.333 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBiBORmKgWpLRAAAuDSaD3Sr6jmtu+gjeN1fQ== 2024-09-13T09:22:55.333 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.1] 2024-09-13T09:22:55.333 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBiBORm9htRMxAAeiZmsX/5XeS/EwxiXN/BaA== 2024-09-13T09:22:55.333 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.2] 2024-09-13T09:22:55.333 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBiBORmC0rQOBAAUmUMs/rptWf+9nXc8ATIOg== 2024-09-13T09:22:55.333 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.3] 2024-09-13T09:22:55.334 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBjBORmyQLKAhAAco0akw60STyoODKx1JSSng== 2024-09-13T09:22:55.334 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.4] 2024-09-13T09:22:55.334 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBjBORmdMFhCBAA8l1j1JCIgkugHx+ESdDAFg== 2024-09-13T09:22:55.334 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.5] 2024-09-13T09:22:55.334 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBjBORmocztDRAApSn+HC77K9PYEC410IOzAQ== 2024-09-13T09:22:55.334 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.6] 2024-09-13T09:22:55.334 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBjBORm+Zp6ExAAXjBxX3TERxd68ppB+dVfow== 2024-09-13T09:22:55.335 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T09:22:55.426 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T09:22:55.477 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T09:22:55.527 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T09:22:55.578 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T09:22:55.631 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T09:22:55.684 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T09:22:55.737 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T09:22:55.790 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T09:22:55.842 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T09:22:55.895 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T09:22:55.948 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T09:22:56.002 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T09:22:56.055 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-13T09:22:56.055 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-09-13T09:22:56.081 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-09-13T09:22:56.175 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-09-13T09:22:56.200 DEBUG:teuthology.orchestra.run.smithi122:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-09-13T09:22:56.255 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-09-13T09:22:56.255 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-09-13T09:22:56.255 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-09-13T09:22:56.298 INFO:tasks.ceph.cluster1.mon.a:Started 2024-09-13T09:22:56.298 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-09-13T09:22:56.299 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-09-13T09:22:56.299 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-09-13T09:22:56.301 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-09-13T09:22:56.301 INFO:tasks.ceph:Setting crush tunables to default 2024-09-13T09:22:56.301 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster cluster1 osd crush tunables default 2024-09-13T09:22:56.470 INFO:teuthology.orchestra.run.smithi122.stderr:adjusted tunables profile to default 2024-09-13T09:22:56.482 INFO:tasks.ceph:check_enable_crimson: False 2024-09-13T09:22:56.482 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-09-13T09:22:56.482 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:22:56.482 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-09-13T09:22:56.507 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:22:56.507 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-09-13T09:22:56.572 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:22:56.572 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-09-13T09:22:56.637 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster cluster1 osd new 6a665a2f-8cfa-47ec-802a-18764b684093 0 2024-09-13T09:22:56.667 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:56.665+0000 7f7de6617040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T09:22:56.710 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:56.709+0000 7f7de6617040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T09:22:56.842 INFO:teuthology.orchestra.run.smithi122.stdout:0 2024-09-13T09:22:56.853 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster cluster1 osd new 4f51e85c-4d33-4de7-9136-233fc8b9e47e 1 2024-09-13T09:22:56.859 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:56.858+0000 7f7de6617040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T09:22:56.911 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:56.910+0000 7f7de6617040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T09:22:57.028 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-09-13T09:22:57.039 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster cluster1 osd new 73635564-c0af-437e-8216-f4bc6a1f4c59 2 2024-09-13T09:22:57.082 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:57.081+0000 7f7de6617040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T09:22:57.198 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:57.196+0000 7f7de6617040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T09:22:57.255 INFO:teuthology.orchestra.run.smithi122.stdout:2 2024-09-13T09:22:57.266 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-09-13T09:22:57.266 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-09-13T09:22:57.309 INFO:tasks.ceph.cluster1.osd.0:Started 2024-09-13T09:22:57.309 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-09-13T09:22:57.309 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-09-13T09:22:57.311 INFO:tasks.ceph.cluster1.osd.1:Started 2024-09-13T09:22:57.311 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-09-13T09:22:57.311 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-09-13T09:22:57.313 INFO:tasks.ceph.cluster1.osd.2:Started 2024-09-13T09:22:57.314 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-13T09:22:57.497 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:22:57.497 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":5,"fsid":"b196829d-0abf-49d7-afed-e85b83e5f85c","created":"2024-09-13T09:22:56.382934+0000","modified":"2024-09-13T09:22:57.253244+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T09:22:57.253244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a665a2f-8cfa-47ec-802a-18764b684093","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4f51e85c-4d33-4de7-9136-233fc8b9e47e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"73635564-c0af-437e-8216-f4bc6a1f4c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T09:22:57.507 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-09-13T09:22:57.507 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-13T09:22:57.539 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:57.538+0000 7f7de6617040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T09:22:57.597 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:57.595+0000 7f7de6617040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T09:22:57.743 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:57.742+0000 7f7de6617040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T09:22:57.810 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-13T09:22:57.910 INFO:tasks.ceph.cluster1.osd.2.smithi122.stderr:2024-09-13T09:22:57.908+0000 7f72ffa31640 -1 Falling back to public interface 2024-09-13T09:22:57.910 INFO:tasks.ceph.cluster1.osd.0.smithi122.stderr:2024-09-13T09:22:57.909+0000 7ff1a92ed640 -1 Falling back to public interface 2024-09-13T09:22:57.926 INFO:tasks.ceph.cluster1.osd.1.smithi122.stderr:2024-09-13T09:22:57.925+0000 7f8867417640 -1 Falling back to public interface 2024-09-13T09:22:57.965 INFO:teuthology.misc.health.smithi122.stdout: 2024-09-13T09:22:57.965 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":5,"fsid":"b196829d-0abf-49d7-afed-e85b83e5f85c","created":"2024-09-13T09:22:56.382934+0000","modified":"2024-09-13T09:22:57.253244+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T09:22:57.253244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a665a2f-8cfa-47ec-802a-18764b684093","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4f51e85c-4d33-4de7-9136-233fc8b9e47e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"73635564-c0af-437e-8216-f4bc6a1f4c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T09:22:57.974 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-13T09:22:58.686 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:58.685+0000 7f7de6617040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T09:22:58.737 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:58.736+0000 7f7de6617040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T09:22:59.098 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:59.097+0000 7f7de6617040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T09:22:59.180 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:59.179+0000 7f7de6617040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T09:22:59.284 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:59.282+0000 7f7de6617040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T09:22:59.329 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:59.328+0000 7f7de6617040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T09:22:59.437 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:59.436+0000 7f7de6617040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T09:22:59.493 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:59.492+0000 7f7de6617040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T09:22:59.537 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:59.535+0000 7f7de6617040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T09:22:59.647 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:59.646+0000 7f7de6617040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T09:22:59.695 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:59.694+0000 7f7de6617040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T09:22:59.745 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:59.744+0000 7f7de6617040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T09:22:59.811 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:59.810+0000 7f7de6617040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T09:22:59.856 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:59.855+0000 7f7de6617040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T09:22:59.899 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:22:59.898+0000 7f7de6617040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T09:23:00.534 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:23:00.533+0000 7f7de6617040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T09:23:01.474 INFO:tasks.ceph.cluster1.osd.1.smithi122.stderr:2024-09-13T09:23:01.473+0000 7f8867417640 -1 osd.1 0 log_to_monitors true 2024-09-13T09:23:01.479 INFO:tasks.ceph.cluster1.osd.0.smithi122.stderr:2024-09-13T09:23:01.478+0000 7ff1a92ed640 -1 osd.0 0 log_to_monitors true 2024-09-13T09:23:01.481 INFO:tasks.ceph.cluster1.osd.2.smithi122.stderr:2024-09-13T09:23:01.480+0000 7f72ffa31640 -1 osd.2 0 log_to_monitors true 2024-09-13T09:23:01.541 INFO:tasks.ceph.cluster1.mgr.x.smithi122.stderr:2024-09-13T09:23:01.540+0000 7f7d86d42640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-13T09:23:03.576 INFO:tasks.ceph.cluster1.osd.0.smithi122.stderr:2024-09-13T09:23:03.575+0000 7ff1a5a91640 -1 osd.0 0 waiting for initial osdmap 2024-09-13T09:23:03.589 INFO:tasks.ceph.cluster1.osd.2.smithi122.stderr:2024-09-13T09:23:03.588+0000 7f72fb9be640 -1 osd.2 0 waiting for initial osdmap 2024-09-13T09:23:03.591 INFO:tasks.ceph.cluster1.osd.0.smithi122.stderr:2024-09-13T09:23:03.590+0000 7ff19d09b640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T09:23:03.605 INFO:tasks.ceph.cluster1.osd.2.smithi122.stderr:2024-09-13T09:23:03.604+0000 7f72f37df640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T09:23:03.615 INFO:tasks.ceph.cluster1.osd.1.smithi122.stderr:2024-09-13T09:23:03.614+0000 7f8862dfe640 -1 osd.1 0 waiting for initial osdmap 2024-09-13T09:23:03.626 INFO:tasks.ceph.cluster1.osd.1.smithi122.stderr:2024-09-13T09:23:03.625+0000 7f885ac1f640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T09:23:04.277 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-13T09:23:04.541 INFO:teuthology.misc.health.smithi122.stdout: 2024-09-13T09:23:04.541 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":7,"fsid":"b196829d-0abf-49d7-afed-e85b83e5f85c","created":"2024-09-13T09:22:56.382934+0000","modified":"2024-09-13T09:23:02.590930+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T09:22:57.253244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a665a2f-8cfa-47ec-802a-18764b684093","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4f51e85c-4d33-4de7-9136-233fc8b9e47e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"73635564-c0af-437e-8216-f4bc6a1f4c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T09:23:04.552 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-13T09:23:10.855 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-13T09:23:11.127 INFO:teuthology.misc.health.smithi122.stdout: 2024-09-13T09:23:11.127 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":12,"fsid":"b196829d-0abf-49d7-afed-e85b83e5f85c","created":"2024-09-13T09:22:56.382934+0000","modified":"2024-09-13T09:23:08.613891+0000","last_up_change":"2024-09-13T09:23:04.592007+0000","last_in_change":"2024-09-13T09:22:57.253244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T09:23:06.568388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6a665a2f-8cfa-47ec-802a-18764b684093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6805","nonce":3789617865},{"type":"v1","addr":"172.21.15.122:6807","nonce":3789617865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6809","nonce":3789617865},{"type":"v1","addr":"172.21.15.122:6811","nonce":3789617865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3789617865},{"type":"v1","addr":"172.21.15.122:6815","nonce":3789617865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3789617865},{"type":"v1","addr":"172.21.15.122:6813","nonce":3789617865}]},"public_addr":"172.21.15.122:6807/3789617865","cluster_addr":"172.21.15.122:6811/3789617865","heartbeat_back_addr":"172.21.15.122:6815/3789617865","heartbeat_front_addr":"172.21.15.122:6813/3789617865","state":["exists","up"]},{"osd":1,"uuid":"4f51e85c-4d33-4de7-9136-233fc8b9e47e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1208258757},{"type":"v1","addr":"172.21.15.122:6817","nonce":1208258757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1208258757},{"type":"v1","addr":"172.21.15.122:6819","nonce":1208258757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1208258757},{"type":"v1","addr":"172.21.15.122:6823","nonce":1208258757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1208258757},{"type":"v1","addr":"172.21.15.122:6821","nonce":1208258757}]},"public_addr":"172.21.15.122:6817/1208258757","cluster_addr":"172.21.15.122:6819/1208258757","heartbeat_back_addr":"172.21.15.122:6823/1208258757","heartbeat_front_addr":"172.21.15.122:6821/1208258757","state":["exists","up"]},{"osd":2,"uuid":"73635564-c0af-437e-8216-f4bc6a1f4c59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1711088402},{"type":"v1","addr":"172.21.15.122:6801","nonce":1711088402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1711088402},{"type":"v1","addr":"172.21.15.122:6803","nonce":1711088402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1711088402},{"type":"v1","addr":"172.21.15.122:6810","nonce":1711088402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1711088402},{"type":"v1","addr":"172.21.15.122:6806","nonce":1711088402}]},"public_addr":"172.21.15.122:6801/1711088402","cluster_addr":"172.21.15.122:6803/1711088402","heartbeat_back_addr":"172.21.15.122:6810/1711088402","heartbeat_front_addr":"172.21.15.122:6806/1711088402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:02.441197+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:02.504333+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:02.520433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T09:23:11.138 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-13T09:23:11.139 INFO:tasks.ceph:Creating RBD pool 2024-09-13T09:23:11.139 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-09-13T09:23:11.632 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'rbd' created 2024-09-13T09:23:11.646 DEBUG:teuthology.orchestra.run.smithi122:> rbd --cluster cluster1 pool init rbd 2024-09-13T09:23:14.653 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-09-13T09:23:14.653 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-09-13T09:23:14.654 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-09-13T09:23:14.654 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-09-13T09:23:14.654 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-13T09:23:14.983 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:14.994 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1876025004},{"type":"v1","addr":"172.21.15.122:6825","nonce":1876025004}]},"active_addr":"172.21.15.122:6825/1876025004","active_change":"2024-09-13T09:23:00.535073+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":1665227582}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":3084559107}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":1743118064}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":584622379}]}]} 2024-09-13T09:23:14.996 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-09-13T09:23:14.996 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-09-13T09:23:14.997 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-13T09:23:15.267 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:15.268 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":16,"fsid":"b196829d-0abf-49d7-afed-e85b83e5f85c","created":"2024-09-13T09:22:56.382934+0000","modified":"2024-09-13T09:23:14.638046+0000","last_up_change":"2024-09-13T09:23:04.592007+0000","last_in_change":"2024-09-13T09:22:57.253244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T09:23:06.568388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T09:23:11.422917+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6a665a2f-8cfa-47ec-802a-18764b684093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6805","nonce":3789617865},{"type":"v1","addr":"172.21.15.122:6807","nonce":3789617865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6809","nonce":3789617865},{"type":"v1","addr":"172.21.15.122:6811","nonce":3789617865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3789617865},{"type":"v1","addr":"172.21.15.122:6815","nonce":3789617865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3789617865},{"type":"v1","addr":"172.21.15.122:6813","nonce":3789617865}]},"public_addr":"172.21.15.122:6807/3789617865","cluster_addr":"172.21.15.122:6811/3789617865","heartbeat_back_addr":"172.21.15.122:6815/3789617865","heartbeat_front_addr":"172.21.15.122:6813/3789617865","state":["exists","up"]},{"osd":1,"uuid":"4f51e85c-4d33-4de7-9136-233fc8b9e47e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1208258757},{"type":"v1","addr":"172.21.15.122:6817","nonce":1208258757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1208258757},{"type":"v1","addr":"172.21.15.122:6819","nonce":1208258757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1208258757},{"type":"v1","addr":"172.21.15.122:6823","nonce":1208258757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1208258757},{"type":"v1","addr":"172.21.15.122:6821","nonce":1208258757}]},"public_addr":"172.21.15.122:6817/1208258757","cluster_addr":"172.21.15.122:6819/1208258757","heartbeat_back_addr":"172.21.15.122:6823/1208258757","heartbeat_front_addr":"172.21.15.122:6821/1208258757","state":["exists","up"]},{"osd":2,"uuid":"73635564-c0af-437e-8216-f4bc6a1f4c59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1711088402},{"type":"v1","addr":"172.21.15.122:6801","nonce":1711088402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1711088402},{"type":"v1","addr":"172.21.15.122:6803","nonce":1711088402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1711088402},{"type":"v1","addr":"172.21.15.122:6810","nonce":1711088402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1711088402},{"type":"v1","addr":"172.21.15.122:6806","nonce":1711088402}]},"public_addr":"172.21.15.122:6801/1711088402","cluster_addr":"172.21.15.122:6803/1711088402","heartbeat_back_addr":"172.21.15.122:6810/1711088402","heartbeat_front_addr":"172.21.15.122:6806/1711088402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:02.441197+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:02.504333+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:02.520433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T09:23:15.280 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-09-13T09:23:15.280 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-13T09:23:15.549 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:15.549 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":16,"fsid":"b196829d-0abf-49d7-afed-e85b83e5f85c","created":"2024-09-13T09:22:56.382934+0000","modified":"2024-09-13T09:23:14.638046+0000","last_up_change":"2024-09-13T09:23:04.592007+0000","last_in_change":"2024-09-13T09:22:57.253244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T09:23:06.568388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T09:23:11.422917+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6a665a2f-8cfa-47ec-802a-18764b684093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6805","nonce":3789617865},{"type":"v1","addr":"172.21.15.122:6807","nonce":3789617865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6809","nonce":3789617865},{"type":"v1","addr":"172.21.15.122:6811","nonce":3789617865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3789617865},{"type":"v1","addr":"172.21.15.122:6815","nonce":3789617865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3789617865},{"type":"v1","addr":"172.21.15.122:6813","nonce":3789617865}]},"public_addr":"172.21.15.122:6807/3789617865","cluster_addr":"172.21.15.122:6811/3789617865","heartbeat_back_addr":"172.21.15.122:6815/3789617865","heartbeat_front_addr":"172.21.15.122:6813/3789617865","state":["exists","up"]},{"osd":1,"uuid":"4f51e85c-4d33-4de7-9136-233fc8b9e47e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1208258757},{"type":"v1","addr":"172.21.15.122:6817","nonce":1208258757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1208258757},{"type":"v1","addr":"172.21.15.122:6819","nonce":1208258757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1208258757},{"type":"v1","addr":"172.21.15.122:6823","nonce":1208258757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1208258757},{"type":"v1","addr":"172.21.15.122:6821","nonce":1208258757}]},"public_addr":"172.21.15.122:6817/1208258757","cluster_addr":"172.21.15.122:6819/1208258757","heartbeat_back_addr":"172.21.15.122:6823/1208258757","heartbeat_front_addr":"172.21.15.122:6821/1208258757","state":["exists","up"]},{"osd":2,"uuid":"73635564-c0af-437e-8216-f4bc6a1f4c59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1711088402},{"type":"v1","addr":"172.21.15.122:6801","nonce":1711088402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1711088402},{"type":"v1","addr":"172.21.15.122:6803","nonce":1711088402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1711088402},{"type":"v1","addr":"172.21.15.122:6810","nonce":1711088402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1711088402},{"type":"v1","addr":"172.21.15.122:6806","nonce":1711088402}]},"public_addr":"172.21.15.122:6801/1711088402","cluster_addr":"172.21.15.122:6803/1711088402","heartbeat_back_addr":"172.21.15.122:6810/1711088402","heartbeat_front_addr":"172.21.15.122:6806/1711088402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:02.441197+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:02.504333+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:02.520433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T09:23:15.562 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-09-13T09:23:15.562 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-09-13T09:23:15.563 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-09-13T09:23:15.700 INFO:teuthology.orchestra.run.smithi122.stdout:34359738373 2024-09-13T09:23:15.700 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-13T09:23:15.701 INFO:teuthology.orchestra.run.smithi122.stdout:34359738374 2024-09-13T09:23:15.701 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-13T09:23:15.702 INFO:teuthology.orchestra.run.smithi122.stdout:34359738374 2024-09-13T09:23:15.702 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-13T09:23:16.025 INFO:teuthology.orchestra.run.smithi122.stdout:34359738372 2024-09-13T09:23:16.031 INFO:teuthology.orchestra.run.smithi122.stdout:34359738372 2024-09-13T09:23:16.031 INFO:teuthology.orchestra.run.smithi122.stdout:34359738371 2024-09-13T09:23:16.038 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738372 for osd.1 2024-09-13T09:23:16.046 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738372 for osd.2 2024-09-13T09:23:16.046 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738371 for osd.0 2024-09-13T09:23:17.039 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-13T09:23:17.046 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-13T09:23:17.047 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-13T09:23:17.337 INFO:teuthology.orchestra.run.smithi122.stdout:34359738374 2024-09-13T09:23:17.350 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.1 2024-09-13T09:23:17.350 DEBUG:teuthology.parallel:result is None 2024-09-13T09:23:17.373 INFO:teuthology.orchestra.run.smithi122.stdout:34359738373 2024-09-13T09:23:17.376 INFO:teuthology.orchestra.run.smithi122.stdout:34359738374 2024-09-13T09:23:17.386 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.0 2024-09-13T09:23:17.387 DEBUG:teuthology.parallel:result is None 2024-09-13T09:23:17.389 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.2 2024-09-13T09:23:17.389 DEBUG:teuthology.parallel:result is None 2024-09-13T09:23:17.389 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-13T09:23:17.390 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-13T09:23:17.696 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:17.696 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-09-13T09:23:17.708 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-13T09:23:16.541412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81576,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033944,"statfs":{"total":289910292480,"available":289826758656,"internally_reserved":0,"allocated":528384,"data_stored":1295395,"data_compressed":59912,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.910486"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:14.650069+0000","last_change":"2024-09-13T09:23:14.650263+0000","last_active":"2024-09-13T09:23:14.650069+0000","last_peered":"2024-09-13T09:23:14.650069+0000","last_clean":"2024-09-13T09:23:14.650069+0000","last_became_active":"2024-09-13T09:23:12.644939+0000","last_became_peered":"2024-09-13T09:23:12.644939+0000","last_unstale":"2024-09-13T09:23:14.650069+0000","last_undegraded":"2024-09-13T09:23:14.650069+0000","last_fullsized":"2024-09-13T09:23:14.650069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:32:52.401407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075243300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:14.649821+0000","last_change":"2024-09-13T09:23:14.650022+0000","last_active":"2024-09-13T09:23:14.649821+0000","last_peered":"2024-09-13T09:23:14.649821+0000","last_clean":"2024-09-13T09:23:14.649821+0000","last_became_active":"2024-09-13T09:23:12.646762+0000","last_became_peered":"2024-09-13T09:23:12.646762+0000","last_unstale":"2024-09-13T09:23:14.649821+0000","last_undegraded":"2024-09-13T09:23:14.649821+0000","last_fullsized":"2024-09-13T09:23:14.649821+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:01:59.548841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051643700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:14.649484+0000","last_change":"2024-09-13T09:23:14.649764+0000","last_active":"2024-09-13T09:23:14.649484+0000","last_peered":"2024-09-13T09:23:14.649484+0000","last_clean":"2024-09-13T09:23:14.649484+0000","last_became_active":"2024-09-13T09:23:12.644747+0000","last_became_peered":"2024-09-13T09:23:12.644747+0000","last_unstale":"2024-09-13T09:23:14.649484+0000","last_undegraded":"2024-09-13T09:23:14.649484+0000","last_fullsized":"2024-09-13T09:23:14.649484+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:59.606001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044772799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:14.649783+0000","last_change":"2024-09-13T09:23:14.650026+0000","last_active":"2024-09-13T09:23:14.649783+0000","last_peered":"2024-09-13T09:23:14.649783+0000","last_clean":"2024-09-13T09:23:14.649783+0000","last_became_active":"2024-09-13T09:23:12.644527+0000","last_became_peered":"2024-09-13T09:23:12.644527+0000","last_unstale":"2024-09-13T09:23:14.649783+0000","last_undegraded":"2024-09-13T09:23:14.649783+0000","last_fullsized":"2024-09-13T09:23:14.649783+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:37:34.744466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071815500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:14.643542+0000","last_change":"2024-09-13T09:23:14.643542+0000","last_active":"2024-09-13T09:23:14.643542+0000","last_peered":"2024-09-13T09:23:14.643542+0000","last_clean":"2024-09-13T09:23:14.643542+0000","last_became_active":"2024-09-13T09:23:12.646192+0000","last_became_peered":"2024-09-13T09:23:12.646192+0000","last_unstale":"2024-09-13T09:23:14.643542+0000","last_undegraded":"2024-09-13T09:23:14.643542+0000","last_fullsized":"2024-09-13T09:23:14.643542+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:48:00.021353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035806400000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:14.982513+0000","last_change":"2024-09-13T09:23:14.982720+0000","last_active":"2024-09-13T09:23:14.982513+0000","last_peered":"2024-09-13T09:23:14.982513+0000","last_clean":"2024-09-13T09:23:14.982513+0000","last_became_active":"2024-09-13T09:23:12.645432+0000","last_became_peered":"2024-09-13T09:23:12.645432+0000","last_unstale":"2024-09-13T09:23:14.982513+0000","last_undegraded":"2024-09-13T09:23:14.982513+0000","last_fullsized":"2024-09-13T09:23:14.982513+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:40:58.609609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036873199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:14.982470+0000","last_change":"2024-09-13T09:23:14.982718+0000","last_active":"2024-09-13T09:23:14.982470+0000","last_peered":"2024-09-13T09:23:14.982470+0000","last_clean":"2024-09-13T09:23:14.982470+0000","last_became_active":"2024-09-13T09:23:12.645717+0000","last_became_peered":"2024-09-13T09:23:12.645717+0000","last_unstale":"2024-09-13T09:23:14.982470+0000","last_undegraded":"2024-09-13T09:23:14.982470+0000","last_fullsized":"2024-09-13T09:23:14.982470+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:55.805671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037606,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:14.649546+0000","last_change":"2024-09-13T09:23:14.649739+0000","last_active":"2024-09-13T09:23:14.649546+0000","last_peered":"2024-09-13T09:23:14.649546+0000","last_clean":"2024-09-13T09:23:14.649546+0000","last_became_active":"2024-09-13T09:23:12.643941+0000","last_became_peered":"2024-09-13T09:23:12.643941+0000","last_unstale":"2024-09-13T09:23:14.649546+0000","last_undegraded":"2024-09-13T09:23:14.649546+0000","last_fullsized":"2024-09-13T09:23:14.649546+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:09.200514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041677400000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":141,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-13T09:23:13.642386+0000","last_change":"2024-09-13T09:23:07.618288+0000","last_active":"2024-09-13T09:23:13.642386+0000","last_peered":"2024-09-13T09:23:13.642386+0000","last_clean":"2024-09-13T09:23:13.642386+0000","last_became_active":"2024-09-13T09:23:07.617395+0000","last_became_peered":"2024-09-13T09:23:07.617395+0000","last_unstale":"2024-09-13T09:23:13.642386+0000","last_undegraded":"2024-09-13T09:23:13.642386+0000","last_fullsized":"2024-09-13T09:23:13.642386+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:06.608062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:06.608062+0000","last_clean_scrub_stamp":"2024-09-13T09:23:06.608062+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:39.950161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59912,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":33720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":631147,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":200704,"data_stored":630528,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T09:23:17.709 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-13T09:23:17.973 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:17.973 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-09-13T09:23:17.984 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-13T09:23:16.541412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81576,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033944,"statfs":{"total":289910292480,"available":289826758656,"internally_reserved":0,"allocated":528384,"data_stored":1295395,"data_compressed":59912,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.910486"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:14.650069+0000","last_change":"2024-09-13T09:23:14.650263+0000","last_active":"2024-09-13T09:23:14.650069+0000","last_peered":"2024-09-13T09:23:14.650069+0000","last_clean":"2024-09-13T09:23:14.650069+0000","last_became_active":"2024-09-13T09:23:12.644939+0000","last_became_peered":"2024-09-13T09:23:12.644939+0000","last_unstale":"2024-09-13T09:23:14.650069+0000","last_undegraded":"2024-09-13T09:23:14.650069+0000","last_fullsized":"2024-09-13T09:23:14.650069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:32:52.401407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075243300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:14.649821+0000","last_change":"2024-09-13T09:23:14.650022+0000","last_active":"2024-09-13T09:23:14.649821+0000","last_peered":"2024-09-13T09:23:14.649821+0000","last_clean":"2024-09-13T09:23:14.649821+0000","last_became_active":"2024-09-13T09:23:12.646762+0000","last_became_peered":"2024-09-13T09:23:12.646762+0000","last_unstale":"2024-09-13T09:23:14.649821+0000","last_undegraded":"2024-09-13T09:23:14.649821+0000","last_fullsized":"2024-09-13T09:23:14.649821+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:01:59.548841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051643700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:14.649484+0000","last_change":"2024-09-13T09:23:14.649764+0000","last_active":"2024-09-13T09:23:14.649484+0000","last_peered":"2024-09-13T09:23:14.649484+0000","last_clean":"2024-09-13T09:23:14.649484+0000","last_became_active":"2024-09-13T09:23:12.644747+0000","last_became_peered":"2024-09-13T09:23:12.644747+0000","last_unstale":"2024-09-13T09:23:14.649484+0000","last_undegraded":"2024-09-13T09:23:14.649484+0000","last_fullsized":"2024-09-13T09:23:14.649484+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:59.606001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044772799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:14.649783+0000","last_change":"2024-09-13T09:23:14.650026+0000","last_active":"2024-09-13T09:23:14.649783+0000","last_peered":"2024-09-13T09:23:14.649783+0000","last_clean":"2024-09-13T09:23:14.649783+0000","last_became_active":"2024-09-13T09:23:12.644527+0000","last_became_peered":"2024-09-13T09:23:12.644527+0000","last_unstale":"2024-09-13T09:23:14.649783+0000","last_undegraded":"2024-09-13T09:23:14.649783+0000","last_fullsized":"2024-09-13T09:23:14.649783+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:37:34.744466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071815500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:14.643542+0000","last_change":"2024-09-13T09:23:14.643542+0000","last_active":"2024-09-13T09:23:14.643542+0000","last_peered":"2024-09-13T09:23:14.643542+0000","last_clean":"2024-09-13T09:23:14.643542+0000","last_became_active":"2024-09-13T09:23:12.646192+0000","last_became_peered":"2024-09-13T09:23:12.646192+0000","last_unstale":"2024-09-13T09:23:14.643542+0000","last_undegraded":"2024-09-13T09:23:14.643542+0000","last_fullsized":"2024-09-13T09:23:14.643542+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:48:00.021353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035806400000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:14.982513+0000","last_change":"2024-09-13T09:23:14.982720+0000","last_active":"2024-09-13T09:23:14.982513+0000","last_peered":"2024-09-13T09:23:14.982513+0000","last_clean":"2024-09-13T09:23:14.982513+0000","last_became_active":"2024-09-13T09:23:12.645432+0000","last_became_peered":"2024-09-13T09:23:12.645432+0000","last_unstale":"2024-09-13T09:23:14.982513+0000","last_undegraded":"2024-09-13T09:23:14.982513+0000","last_fullsized":"2024-09-13T09:23:14.982513+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:40:58.609609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036873199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:14.982470+0000","last_change":"2024-09-13T09:23:14.982718+0000","last_active":"2024-09-13T09:23:14.982470+0000","last_peered":"2024-09-13T09:23:14.982470+0000","last_clean":"2024-09-13T09:23:14.982470+0000","last_became_active":"2024-09-13T09:23:12.645717+0000","last_became_peered":"2024-09-13T09:23:12.645717+0000","last_unstale":"2024-09-13T09:23:14.982470+0000","last_undegraded":"2024-09-13T09:23:14.982470+0000","last_fullsized":"2024-09-13T09:23:14.982470+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:55.805671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037606,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:14.649546+0000","last_change":"2024-09-13T09:23:14.649739+0000","last_active":"2024-09-13T09:23:14.649546+0000","last_peered":"2024-09-13T09:23:14.649546+0000","last_clean":"2024-09-13T09:23:14.649546+0000","last_became_active":"2024-09-13T09:23:12.643941+0000","last_became_peered":"2024-09-13T09:23:12.643941+0000","last_unstale":"2024-09-13T09:23:14.649546+0000","last_undegraded":"2024-09-13T09:23:14.649546+0000","last_fullsized":"2024-09-13T09:23:14.649546+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:09.200514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041677400000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":141,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-13T09:23:13.642386+0000","last_change":"2024-09-13T09:23:07.618288+0000","last_active":"2024-09-13T09:23:13.642386+0000","last_peered":"2024-09-13T09:23:13.642386+0000","last_clean":"2024-09-13T09:23:13.642386+0000","last_became_active":"2024-09-13T09:23:07.617395+0000","last_became_peered":"2024-09-13T09:23:07.617395+0000","last_unstale":"2024-09-13T09:23:13.642386+0000","last_undegraded":"2024-09-13T09:23:13.642386+0000","last_fullsized":"2024-09-13T09:23:13.642386+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:06.608062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:06.608062+0000","last_clean_scrub_stamp":"2024-09-13T09:23:06.608062+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:39.950161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59912,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":33720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":631147,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":200704,"data_stored":630528,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T09:23:17.985 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-13T09:23:17.985 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-09-13T09:23:17.986 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-09-13T09:23:17.986 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-09-13T09:23:18.288 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:18.288 INFO:teuthology.orchestra.run.smithi122.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T09:23:18.300 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-09-13T09:23:18.301 INFO:teuthology.run_tasks:Running task ceph... 2024-09-13T09:23:18.309 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-09-13T09:23:18.309 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-13T09:23:18.309 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7901034', 'branch': 'reef', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7901034', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': '7a853b1706d076d58e0da7a5d3b9b9eb36276880'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 248, 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412', 'sleep_before_teardown': 0, 'subset': '116/128', 'suite': 'rbd', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_7a853b1706d076d58e0da7a5d3b9b9eb36276880/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '7a853b1706d076d58e0da7a5d3b9b9eb36276880', 'targets': {'smithi122.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBPLTJok4PZJYDU2rwScvJ7rB23VRq1g0ehgis1fGIuL1AVwW47oni1FTJmM56obzAHWh8Hf7+Ni6NVwEMRUVuA=', 'smithi190.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD/2hn0Bf/KzJKwZT28wcQLhGlXAxiKjAtWMHBFSyYnSZxSL3Z81UbS2xmNOY2cVn5onuYGXxBkdym2Qn+M0/v8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_22:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-13T09:23:18.310 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-13T09:23:18.312 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-13T09:23:18.331 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T09:23:18.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T09:23:18.401 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:23:18.402 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T09:23:18.461 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T09:23:18.461 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_1 2024-09-13T09:23:18.519 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T09:23:18.520 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T09:23:18.520 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-13T09:23:18.520 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T09:23:18.520 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T09:23:18.520 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-13 09:22:28.061002453 +0000 2024-09-13T09:23:18.520 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-13 09:20:08.737127335 +0000 2024-09-13T09:23:18.520 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-13 09:20:08.737127335 +0000 2024-09-13T09:23:18.520 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-09-13 09:20:08.737127335 +0000 2024-09-13T09:23:18.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T09:23:18.587 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-13T09:23:18.587 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-13T09:23:18.587 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000202574 s, 2.5 MB/s 2024-09-13T09:23:18.589 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T09:23:18.648 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_2 2024-09-13T09:23:18.705 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T09:23:18.705 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T09:23:18.705 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-13T09:23:18.706 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T09:23:18.706 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T09:23:18.706 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-13 09:22:28.062002424 +0000 2024-09-13T09:23:18.706 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-13 09:20:09.062117581 +0000 2024-09-13T09:23:18.706 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-13 09:20:09.062117581 +0000 2024-09-13T09:23:18.706 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-09-13 09:20:09.062117581 +0000 2024-09-13T09:23:18.706 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T09:23:18.774 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-13T09:23:18.774 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-13T09:23:18.774 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.00019513 s, 2.6 MB/s 2024-09-13T09:23:18.776 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T09:23:18.833 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_3 2024-09-13T09:23:18.889 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T09:23:18.889 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T09:23:18.889 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-13T09:23:18.889 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T09:23:18.889 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T09:23:18.889 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-13 09:22:28.062002424 +0000 2024-09-13T09:23:18.889 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-13 09:20:09.383107945 +0000 2024-09-13T09:23:18.890 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-13 09:20:09.383107945 +0000 2024-09-13T09:23:18.890 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-09-13 09:20:09.383107945 +0000 2024-09-13T09:23:18.890 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T09:23:18.953 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-13T09:23:18.953 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-13T09:23:18.953 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000215955 s, 2.4 MB/s 2024-09-13T09:23:18.954 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T09:23:19.013 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_4 2024-09-13T09:23:19.069 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T09:23:19.069 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T09:23:19.069 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-13T09:23:19.069 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T09:23:19.069 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T09:23:19.069 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-13 09:22:28.062002424 +0000 2024-09-13T09:23:19.069 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-13 09:20:09.711098100 +0000 2024-09-13T09:23:19.069 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-13 09:20:09.711098100 +0000 2024-09-13T09:23:19.069 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-09-13 09:20:09.711098100 +0000 2024-09-13T09:23:19.070 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T09:23:19.132 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-13T09:23:19.133 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-13T09:23:19.133 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000187422 s, 2.7 MB/s 2024-09-13T09:23:19.135 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T09:23:19.191 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T09:23:19.191 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-13T09:23:19.191 INFO:tasks.ceph:Generating config... 2024-09-13T09:23:19.193 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-13T09:23:19.193 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-13T09:23:19.193 INFO:tasks.ceph:[client] log to stderr = False 2024-09-13T09:23:19.193 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-13T09:23:19.193 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-09-13T09:23:19.193 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:23:19.193 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:23:19.193 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:23:19.193 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:23:19.193 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:23:19.193 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:23:19.194 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:23:19.194 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:23:19.194 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-13T09:23:19.194 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-13T09:23:19.194 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-13T09:23:19.194 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-13T09:23:19.194 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-13T09:23:19.194 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-13T09:23:19.194 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-13T09:23:19.194 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-13T09:23:19.194 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-13T09:23:19.194 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-13T09:23:19.195 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-13T09:23:19.195 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-13T09:23:19.195 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-13T09:23:19.195 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-13T09:23:19.195 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-13T09:23:19.195 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-13T09:23:19.195 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-13T09:23:19.195 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-13T09:23:19.195 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-13T09:23:19.195 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-13T09:23:19.195 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-13T09:23:19.195 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-13T09:23:19.196 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-09-13T09:23:19.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-09-13T09:23:19.271 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster2.keyring 2024-09-13T09:23:19.273 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-09-13T09:23:19.356 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-13T09:23:19.421 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.190')] 2024-09-13T09:23:19.421 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.190', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-13T09:23:19.422 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:23:19.422 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T09:23:19.476 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.190 --print /home/ubuntu/cephtest/cluster2.monmap 2024-09-13T09:23:19.555 INFO:teuthology.orchestra.run.smithi190.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-09-13T09:23:19.555 INFO:teuthology.orchestra.run.smithi190.stdout:monmaptool: generated fsid 4884b31c-ac19-4f07-b0a6-5fdd65ee73f4 2024-09-13T09:23:19.556 INFO:teuthology.orchestra.run.smithi190.stdout:setting min_mon_release = pacific 2024-09-13T09:23:19.556 INFO:teuthology.orchestra.run.smithi190.stdout:epoch 0 2024-09-13T09:23:19.556 INFO:teuthology.orchestra.run.smithi190.stdout:fsid 4884b31c-ac19-4f07-b0a6-5fdd65ee73f4 2024-09-13T09:23:19.556 INFO:teuthology.orchestra.run.smithi190.stdout:last_changed 2024-09-13T09:23:19.555248+0000 2024-09-13T09:23:19.556 INFO:teuthology.orchestra.run.smithi190.stdout:created 2024-09-13T09:23:19.555248+0000 2024-09-13T09:23:19.556 INFO:teuthology.orchestra.run.smithi190.stdout:min_mon_release 16 (pacific) 2024-09-13T09:23:19.556 INFO:teuthology.orchestra.run.smithi190.stdout:election_strategy: 1 2024-09-13T09:23:19.556 INFO:teuthology.orchestra.run.smithi190.stdout:0: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.a 2024-09-13T09:23:19.556 INFO:teuthology.orchestra.run.smithi190.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-09-13T09:23:19.557 DEBUG:teuthology.orchestra.run.smithi190:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T09:23:19.612 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 4884b31c-ac19-4f07-b0a6-5fdd65ee73f4... 2024-09-13T09:23:19.613 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-13T09:23:19.616 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-13T09:23:19.663 INFO:teuthology.orchestra.run.smithi122.stdout:[global] 2024-09-13T09:23:19.663 INFO:teuthology.orchestra.run.smithi122.stdout: chdir = "" 2024-09-13T09:23:19.663 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T09:23:19.663 INFO:teuthology.orchestra.run.smithi122.stdout: auth supported = cephx 2024-09-13T09:23:19.663 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.663 INFO:teuthology.orchestra.run.smithi122.stdout: filestore xattr use omap = true 2024-09-13T09:23:19.664 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.664 INFO:teuthology.orchestra.run.smithi122.stdout: mon clock drift allowed = 1.000 2024-09-13T09:23:19.664 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.664 INFO:teuthology.orchestra.run.smithi122.stdout: osd crush chooseleaf type = 0 2024-09-13T09:23:19.664 INFO:teuthology.orchestra.run.smithi122.stdout: auth debug = true 2024-09-13T09:23:19.664 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.664 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on old message = true 2024-09-13T09:23:19.664 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on bug = true 2024-09-13T09:23:19.664 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.664 INFO:teuthology.orchestra.run.smithi122.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T09:23:19.664 INFO:teuthology.orchestra.run.smithi122.stdout: mon pg warn max object skew = 0 2024-09-13T09:23:19.664 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.664 INFO:teuthology.orchestra.run.smithi122.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T09:23:19.664 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T09:23:19.664 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.665 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default size = 2 2024-09-13T09:23:19.665 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.665 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow primary affinity = true 2024-09-13T09:23:19.665 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow pg remap = true 2024-09-13T09:23:19.665 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on legacy crush tunables = false 2024-09-13T09:23:19.665 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on crush straw calc version zero = false 2024-09-13T09:23:19.665 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on no sortbitwise = false 2024-09-13T09:23:19.665 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on osd down out interval zero = false 2024-09-13T09:23:19.665 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on too few osds = false 2024-09-13T09:23:19.665 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T09:23:19.665 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T09:23:19.665 INFO:teuthology.orchestra.run.smithi122.stdout: mon_allow_pool_size_one = true 2024-09-13T09:23:19.665 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.666 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T09:23:19.666 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.666 INFO:teuthology.orchestra.run.smithi122.stdout: osd default data pool replay window = 5 2024-09-13T09:23:19.666 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.666 INFO:teuthology.orchestra.run.smithi122.stdout: mon allow pool delete = true 2024-09-13T09:23:19.666 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.666 INFO:teuthology.orchestra.run.smithi122.stdout: mon cluster log file level = debug 2024-09-13T09:23:19.666 INFO:teuthology.orchestra.run.smithi122.stdout: debug asserts on shutdown = true 2024-09-13T09:23:19.666 INFO:teuthology.orchestra.run.smithi122.stdout: mon health detail to clog = false 2024-09-13T09:23:19.666 INFO:teuthology.orchestra.run.smithi122.stdout: mon host = 172.21.15.190 2024-09-13T09:23:19.666 INFO:teuthology.orchestra.run.smithi122.stdout: mon client directed command retry = 5 2024-09-13T09:23:19.666 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on pool no app = False 2024-09-13T09:23:19.666 INFO:teuthology.orchestra.run.smithi122.stdout: ms inject socket failures = 5000 2024-09-13T09:23:19.667 INFO:teuthology.orchestra.run.smithi122.stdout: fsid = 4884b31c-ac19-4f07-b0a6-5fdd65ee73f4 2024-09-13T09:23:19.667 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.667 INFO:teuthology.orchestra.run.smithi122.stdout:[osd] 2024-09-13T09:23:19.667 INFO:teuthology.orchestra.run.smithi122.stdout: osd journal size = 100 2024-09-13T09:23:19.667 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.667 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub load threshold = 5.0 2024-09-13T09:23:19.667 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub max interval = 600 2024-09-13T09:23:19.667 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock profile = high_recovery_ops 2024-09-13T09:23:19.667 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.667 INFO:teuthology.orchestra.run.smithi122.stdout: osd recover clone overlap = true 2024-09-13T09:23:19.667 INFO:teuthology.orchestra.run.smithi122.stdout: osd recovery max chunk = 1048576 2024-09-13T09:23:19.667 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.667 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug shutdown = true 2024-09-13T09:23:19.667 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug op order = true 2024-09-13T09:23:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify stray on activate = true 2024-09-13T09:23:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: osd open classes on start = true 2024-09-13T09:23:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug pg log writeout = true 2024-09-13T09:23:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: osd deep scrub update digest min age = 30 2024-09-13T09:23:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: osd map max advance = 10 2024-09-13T09:23:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: journal zero on create = true 2024-09-13T09:23:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: filestore ondisk finisher threads = 3 2024-09-13T09:23:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: filestore apply finisher threads = 3 2024-09-13T09:23:19.668 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: bdev debug aio = true 2024-09-13T09:23:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug misdirected ops = true 2024-09-13T09:23:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore block size = 96636764160 2024-09-13T09:23:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore compression algorithm = snappy 2024-09-13T09:23:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore compression mode = aggressive 2024-09-13T09:23:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore fsck on mount = True 2024-09-13T09:23:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluefs = 1/20 2024-09-13T09:23:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluestore = 1/20 2024-09-13T09:23:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-09-13T09:23:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 20 2024-09-13T09:23:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: debug rocksdb = 4/10 2024-09-13T09:23:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T09:23:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd full ratio = 0.9 2024-09-13T09:23:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd nearfull ratio = 0.8 2024-09-13T09:23:19.669 INFO:teuthology.orchestra.run.smithi122.stdout: osd failsafe full ratio = 0.95 2024-09-13T09:23:19.670 INFO:teuthology.orchestra.run.smithi122.stdout: osd objectstore = bluestore 2024-09-13T09:23:19.670 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.670 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr] 2024-09-13T09:23:19.670 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-09-13T09:23:19.670 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 20 2024-09-13T09:23:19.670 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-09-13T09:23:19.670 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-09-13T09:23:19.670 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-09-13T09:23:19.670 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-09-13T09:23:19.670 INFO:teuthology.orchestra.run.smithi122.stdout: mgr/telemetry/nag = false 2024-09-13T09:23:19.670 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.670 INFO:teuthology.orchestra.run.smithi122.stdout:[mon] 2024-09-13T09:23:19.670 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-09-13T09:23:19.670 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-09-13T09:23:19.671 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 20 2024-09-13T09:23:19.671 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-09-13T09:23:19.671 INFO:teuthology.orchestra.run.smithi122.stdout: mon data avail warn = 5 2024-09-13T09:23:19.671 INFO:teuthology.orchestra.run.smithi122.stdout: mon mgr mkfs grace = 240 2024-09-13T09:23:19.671 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-09-13T09:23:19.671 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd reporter subtree level = osd 2024-09-13T09:23:19.671 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd prime pg temp = true 2024-09-13T09:23:19.671 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-09-13T09:23:19.671 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.671 INFO:teuthology.orchestra.run.smithi122.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T09:23:19.671 INFO:teuthology.orchestra.run.smithi122.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T09:23:19.671 INFO:teuthology.orchestra.run.smithi122.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T09:23:19.671 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.671 INFO:teuthology.orchestra.run.smithi122.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T09:23:19.671 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T09:23:19.672 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T09:23:19.672 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.672 INFO:teuthology.orchestra.run.smithi122.stdout: # 1m isn't quite enough 2024-09-13T09:23:19.672 INFO:teuthology.orchestra.run.smithi122.stdout: mon_down_mkfs_grace = 2m 2024-09-13T09:23:19.672 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.672 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_filestore_osds = false 2024-09-13T09:23:19.672 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T09:23:19.672 INFO:teuthology.orchestra.run.smithi122.stdout:[client] 2024-09-13T09:23:19.672 INFO:teuthology.orchestra.run.smithi122.stdout: rgw cache enabled = true 2024-09-13T09:23:19.672 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable ops log = true 2024-09-13T09:23:19.672 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable usage log = true 2024-09-13T09:23:19.672 INFO:teuthology.orchestra.run.smithi122.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T09:23:19.673 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T09:23:19.673 INFO:teuthology.orchestra.run.smithi122.stdout: debug rbd = 20 2024-09-13T09:23:19.673 INFO:teuthology.orchestra.run.smithi122.stdout: debug rbd_mirror = 15 2024-09-13T09:23:19.673 INFO:teuthology.orchestra.run.smithi122.stdout: log to stderr = False 2024-09-13T09:23:19.673 INFO:teuthology.orchestra.run.smithi122.stdout: rbd default features = 125 2024-09-13T09:23:19.673 INFO:teuthology.orchestra.run.smithi122.stdout: rbd mirror image policy type = simple 2024-09-13T09:23:19.673 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.a] 2024-09-13T09:23:19.673 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.0] 2024-09-13T09:23:19.673 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:23:19.673 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:23:19.673 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.1] 2024-09-13T09:23:19.673 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:23:19.673 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:23:19.673 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.2] 2024-09-13T09:23:19.673 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:23:19.674 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:23:19.674 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.3] 2024-09-13T09:23:19.674 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:23:19.674 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:23:19.695 INFO:teuthology.orchestra.run.smithi190.stdout:[global] 2024-09-13T09:23:19.695 INFO:teuthology.orchestra.run.smithi190.stdout: chdir = "" 2024-09-13T09:23:19.695 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T09:23:19.695 INFO:teuthology.orchestra.run.smithi190.stdout: auth supported = cephx 2024-09-13T09:23:19.696 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.696 INFO:teuthology.orchestra.run.smithi190.stdout: filestore xattr use omap = true 2024-09-13T09:23:19.696 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.696 INFO:teuthology.orchestra.run.smithi190.stdout: mon clock drift allowed = 1.000 2024-09-13T09:23:19.696 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.696 INFO:teuthology.orchestra.run.smithi190.stdout: osd crush chooseleaf type = 0 2024-09-13T09:23:19.696 INFO:teuthology.orchestra.run.smithi190.stdout: auth debug = true 2024-09-13T09:23:19.696 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.696 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on old message = true 2024-09-13T09:23:19.696 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on bug = true 2024-09-13T09:23:19.696 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.696 INFO:teuthology.orchestra.run.smithi190.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T09:23:19.696 INFO:teuthology.orchestra.run.smithi190.stdout: mon pg warn max object skew = 0 2024-09-13T09:23:19.697 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.697 INFO:teuthology.orchestra.run.smithi190.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T09:23:19.697 INFO:teuthology.orchestra.run.smithi190.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T09:23:19.697 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.697 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default size = 2 2024-09-13T09:23:19.697 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.697 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow primary affinity = true 2024-09-13T09:23:19.697 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow pg remap = true 2024-09-13T09:23:19.697 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on legacy crush tunables = false 2024-09-13T09:23:19.697 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on crush straw calc version zero = false 2024-09-13T09:23:19.697 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on no sortbitwise = false 2024-09-13T09:23:19.697 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on osd down out interval zero = false 2024-09-13T09:23:19.697 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on too few osds = false 2024-09-13T09:23:19.697 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T09:23:19.698 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T09:23:19.698 INFO:teuthology.orchestra.run.smithi190.stdout: mon_allow_pool_size_one = true 2024-09-13T09:23:19.698 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.698 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T09:23:19.698 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.698 INFO:teuthology.orchestra.run.smithi190.stdout: osd default data pool replay window = 5 2024-09-13T09:23:19.698 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.698 INFO:teuthology.orchestra.run.smithi190.stdout: mon allow pool delete = true 2024-09-13T09:23:19.698 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.698 INFO:teuthology.orchestra.run.smithi190.stdout: mon cluster log file level = debug 2024-09-13T09:23:19.698 INFO:teuthology.orchestra.run.smithi190.stdout: debug asserts on shutdown = true 2024-09-13T09:23:19.698 INFO:teuthology.orchestra.run.smithi190.stdout: mon health detail to clog = false 2024-09-13T09:23:19.698 INFO:teuthology.orchestra.run.smithi190.stdout: mon host = 172.21.15.190 2024-09-13T09:23:19.698 INFO:teuthology.orchestra.run.smithi190.stdout: mon client directed command retry = 5 2024-09-13T09:23:19.699 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on pool no app = False 2024-09-13T09:23:19.699 INFO:teuthology.orchestra.run.smithi190.stdout: ms inject socket failures = 5000 2024-09-13T09:23:19.699 INFO:teuthology.orchestra.run.smithi190.stdout: fsid = 4884b31c-ac19-4f07-b0a6-5fdd65ee73f4 2024-09-13T09:23:19.699 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.699 INFO:teuthology.orchestra.run.smithi190.stdout:[osd] 2024-09-13T09:23:19.699 INFO:teuthology.orchestra.run.smithi190.stdout: osd journal size = 100 2024-09-13T09:23:19.699 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.699 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub load threshold = 5.0 2024-09-13T09:23:19.699 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub max interval = 600 2024-09-13T09:23:19.699 INFO:teuthology.orchestra.run.smithi190.stdout: osd mclock profile = high_recovery_ops 2024-09-13T09:23:19.699 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.699 INFO:teuthology.orchestra.run.smithi190.stdout: osd recover clone overlap = true 2024-09-13T09:23:19.699 INFO:teuthology.orchestra.run.smithi190.stdout: osd recovery max chunk = 1048576 2024-09-13T09:23:19.699 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.699 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug shutdown = true 2024-09-13T09:23:19.700 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug op order = true 2024-09-13T09:23:19.700 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify stray on activate = true 2024-09-13T09:23:19.700 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.700 INFO:teuthology.orchestra.run.smithi190.stdout: osd open classes on start = true 2024-09-13T09:23:19.700 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug pg log writeout = true 2024-09-13T09:23:19.700 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.700 INFO:teuthology.orchestra.run.smithi190.stdout: osd deep scrub update digest min age = 30 2024-09-13T09:23:19.700 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.700 INFO:teuthology.orchestra.run.smithi190.stdout: osd map max advance = 10 2024-09-13T09:23:19.700 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.700 INFO:teuthology.orchestra.run.smithi190.stdout: journal zero on create = true 2024-09-13T09:23:19.700 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.700 INFO:teuthology.orchestra.run.smithi190.stdout: filestore ondisk finisher threads = 3 2024-09-13T09:23:19.701 INFO:teuthology.orchestra.run.smithi190.stdout: filestore apply finisher threads = 3 2024-09-13T09:23:19.701 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.701 INFO:teuthology.orchestra.run.smithi190.stdout: bdev debug aio = true 2024-09-13T09:23:19.701 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug misdirected ops = true 2024-09-13T09:23:19.701 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore block size = 96636764160 2024-09-13T09:23:19.701 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore compression algorithm = snappy 2024-09-13T09:23:19.701 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore compression mode = aggressive 2024-09-13T09:23:19.701 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore fsck on mount = True 2024-09-13T09:23:19.701 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluefs = 1/20 2024-09-13T09:23:19.701 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluestore = 1/20 2024-09-13T09:23:19.701 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-13T09:23:19.701 INFO:teuthology.orchestra.run.smithi190.stdout: debug osd = 20 2024-09-13T09:23:19.701 INFO:teuthology.orchestra.run.smithi190.stdout: debug rocksdb = 4/10 2024-09-13T09:23:19.701 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T09:23:19.702 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd full ratio = 0.9 2024-09-13T09:23:19.702 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd nearfull ratio = 0.8 2024-09-13T09:23:19.702 INFO:teuthology.orchestra.run.smithi190.stdout: osd failsafe full ratio = 0.95 2024-09-13T09:23:19.702 INFO:teuthology.orchestra.run.smithi190.stdout: osd objectstore = bluestore 2024-09-13T09:23:19.702 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.702 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr] 2024-09-13T09:23:19.702 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-13T09:23:19.702 INFO:teuthology.orchestra.run.smithi190.stdout: debug mgr = 20 2024-09-13T09:23:19.702 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2024-09-13T09:23:19.702 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2024-09-13T09:23:19.702 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2024-09-13T09:23:19.702 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2024-09-13T09:23:19.703 INFO:teuthology.orchestra.run.smithi190.stdout: mgr/telemetry/nag = false 2024-09-13T09:23:19.703 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.703 INFO:teuthology.orchestra.run.smithi190.stdout:[mon] 2024-09-13T09:23:19.703 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-13T09:23:19.703 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2024-09-13T09:23:19.703 INFO:teuthology.orchestra.run.smithi190.stdout: debug paxos = 20 2024-09-13T09:23:19.703 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2024-09-13T09:23:19.703 INFO:teuthology.orchestra.run.smithi190.stdout: mon data avail warn = 5 2024-09-13T09:23:19.703 INFO:teuthology.orchestra.run.smithi190.stdout: mon mgr mkfs grace = 240 2024-09-13T09:23:19.703 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2024-09-13T09:23:19.703 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd reporter subtree level = osd 2024-09-13T09:23:19.703 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd prime pg temp = true 2024-09-13T09:23:19.703 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2024-09-13T09:23:19.703 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.703 INFO:teuthology.orchestra.run.smithi190.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T09:23:19.703 INFO:teuthology.orchestra.run.smithi190.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T09:23:19.704 INFO:teuthology.orchestra.run.smithi190.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T09:23:19.704 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.704 INFO:teuthology.orchestra.run.smithi190.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T09:23:19.704 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T09:23:19.704 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T09:23:19.704 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.704 INFO:teuthology.orchestra.run.smithi190.stdout: # 1m isn't quite enough 2024-09-13T09:23:19.704 INFO:teuthology.orchestra.run.smithi190.stdout: mon_down_mkfs_grace = 2m 2024-09-13T09:23:19.704 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.704 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_filestore_osds = false 2024-09-13T09:23:19.704 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:19.704 INFO:teuthology.orchestra.run.smithi190.stdout:[client] 2024-09-13T09:23:19.704 INFO:teuthology.orchestra.run.smithi190.stdout: rgw cache enabled = true 2024-09-13T09:23:19.704 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable ops log = true 2024-09-13T09:23:19.704 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable usage log = true 2024-09-13T09:23:19.704 INFO:teuthology.orchestra.run.smithi190.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T09:23:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T09:23:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: debug rbd = 20 2024-09-13T09:23:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: debug rbd_mirror = 15 2024-09-13T09:23:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: log to stderr = False 2024-09-13T09:23:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: rbd default features = 125 2024-09-13T09:23:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: rbd mirror image policy type = simple 2024-09-13T09:23:19.705 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.a] 2024-09-13T09:23:19.705 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.0] 2024-09-13T09:23:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:23:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:23:19.705 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.1] 2024-09-13T09:23:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:23:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:23:19.706 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.2] 2024-09-13T09:23:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:23:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:23:19.706 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.3] 2024-09-13T09:23:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-13T09:23:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-13T09:23:19.707 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-09-13T09:23:19.707 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-09-13T09:23:19.801 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-13T09:23:19.802 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:23:19.802 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-09-13T09:23:19.818 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:23:19.818 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-09-13T09:23:19.873 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T09:23:19.873 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:23:19.873 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-13T09:23:19.873 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-13T09:23:19.913 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:23:19.913 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-13T09:23:19.970 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi190.front.sepia.ceph.com 2024-09-13T09:23:19.971 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:23:19.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-13T09:23:19.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-13T09:23:20.006 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:23:20.006 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-13T09:23:20.062 INFO:tasks.ceph:Setting up mon nodes... 2024-09-13T09:23:20.062 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-13T09:23:20.062 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-13T09:23:20.118 INFO:teuthology.orchestra.run.smithi122.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-13T09:23:20.121 INFO:tasks.ceph:Setting up mds nodes... 2024-09-13T09:23:20.121 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-13T09:23:20.122 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-09-13T09:23:20.164 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-09-13T09:23:20.178 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-09-13T09:23:20.219 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-09-13T09:23:20.231 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-13T09:23:20.312 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-13T09:23:20.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-13T09:23:20.404 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-13T09:23:20.417 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-13T09:23:20.497 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-13T09:23:20.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-13T09:23:20.590 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-13T09:23:20.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-13T09:23:20.683 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-13T09:23:20.696 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-13T09:23:20.777 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-13T09:23:20.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-13T09:23:20.870 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-13T09:23:20.882 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-13T09:23:20.882 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi122.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-13T09:23:20.883 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-09-13T09:23:20.946 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T09:23:20.947 INFO:tasks.ceph:role: cluster2.osd.0 2024-09-13T09:23:20.947 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-13T09:23:20.947 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T09:23:21.019 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T09:23:21.019 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T09:23:21.019 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T09:23:21.019 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T09:23:21.019 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T09:23:21.020 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-09-13T09:23:21.020 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T09:23:21.020 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T09:23:21.020 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T09:23:21.020 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T09:23:21.528 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-09-13T09:23:21.529 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-09-13T09:23:21.530 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-09-13T09:23:21.657 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-09-13T09:23:21.690 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-09-13T09:23:21.755 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T09:23:21.755 INFO:tasks.ceph:role: cluster2.osd.1 2024-09-13T09:23:21.755 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-13T09:23:21.755 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T09:23:21.828 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T09:23:21.828 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T09:23:21.828 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T09:23:21.828 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T09:23:21.828 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T09:23:21.829 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-09-13T09:23:21.829 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T09:23:21.829 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T09:23:21.829 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T09:23:21.829 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T09:23:22.295 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-09-13T09:23:22.297 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-09-13T09:23:22.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-09-13T09:23:22.393 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-09-13T09:23:22.422 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-09-13T09:23:22.488 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T09:23:22.488 INFO:tasks.ceph:role: cluster2.osd.2 2024-09-13T09:23:22.488 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-13T09:23:22.488 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T09:23:22.561 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T09:23:22.561 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T09:23:22.561 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T09:23:22.561 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T09:23:22.561 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T09:23:22.562 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-09-13T09:23:22.562 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T09:23:22.562 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T09:23:22.562 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T09:23:22.562 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T09:23:23.062 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-09-13T09:23:23.064 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-09-13T09:23:23.064 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-09-13T09:23:23.182 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-09-13T09:23:23.214 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-13T09:23:23.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-13T09:23:23.303+0000 7faaa16f1640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-09-13T09:23:23.311 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-13T09:23:23.309+0000 7faaa16f1640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-09-13T09:23:23.311 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-13T09:23:23.309+0000 7faaa16f1640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-09-13T09:23:23.311 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-13T09:23:23.309+0000 7faaa16f1640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-09-13T09:23:23.311 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-13T09:23:23.309+0000 7faaa16f1640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-09-13T09:23:23.312 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-13T09:23:23.311+0000 7faaa16f1640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-09-13T09:23:26.173 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-09-13T09:23:26.201 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-13T09:23:26.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-13T09:23:26.288+0000 7f1d5bb29640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-09-13T09:23:26.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-13T09:23:26.288+0000 7f1d5bb29640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-09-13T09:23:26.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-13T09:23:26.288+0000 7f1d5bb29640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-09-13T09:23:26.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-13T09:23:26.288+0000 7f1d5bb29640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-09-13T09:23:26.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-13T09:23:26.288+0000 7f1d5bb29640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-09-13T09:23:26.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-13T09:23:26.289+0000 7f1d5bb29640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-09-13T09:23:29.141 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-09-13T09:23:29.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-13T09:23:29.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-13T09:23:29.258+0000 7f507be0d640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-09-13T09:23:29.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-13T09:23:29.258+0000 7f507be0d640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-09-13T09:23:29.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-13T09:23:29.258+0000 7f507be0d640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-09-13T09:23:29.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-13T09:23:29.258+0000 7f507be0d640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-09-13T09:23:29.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-13T09:23:29.258+0000 7f507be0d640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-09-13T09:23:29.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-13T09:23:29.259+0000 7f507be0d640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-09-13T09:23:32.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-09-13T09:23:32.149 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-13T09:23:32.149 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:23:32.149 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-09-13T09:23:32.178 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:23:32.178 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-09-13T09:23:32.217 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:23:32.217 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-09-13T09:23:32.282 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:23:32.282 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-09-13T09:23:32.348 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T09:23:32.348 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-09-13T09:23:32.367 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:23:32.367 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-09-13T09:23:32.414 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:23:32.414 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-09-13T09:23:32.470 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:23:32.470 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-09-13T09:23:32.525 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:23:32.525 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-09-13T09:23:32.580 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:23:32.580 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-09-13T09:23:32.635 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:23:32.635 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-09-13T09:23:32.690 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:23:32.690 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-09-13T09:23:32.745 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:23:32.745 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-09-13T09:23:32.801 INFO:tasks.ceph:Adding keys to all mons... 2024-09-13T09:23:32.801 DEBUG:teuthology.orchestra.run.smithi190:> sudo tee -a /etc/ceph/cluster2.keyring 2024-09-13T09:23:32.864 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr.x] 2024-09-13T09:23:32.864 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCIBORmWgARBxAAyV119p/CJzkDVrHMbrYjQQ== 2024-09-13T09:23:32.864 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.0] 2024-09-13T09:23:32.864 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCLBORmdP8qEhAAgOQ8ogspX1P+Tm9cppl4JA== 2024-09-13T09:23:32.864 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.1] 2024-09-13T09:23:32.864 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCOBORm/iU8ERAA51WJYhFWQLMHldq1ot/VXw== 2024-09-13T09:23:32.864 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.2] 2024-09-13T09:23:32.865 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCRBORmAeF5DxAA+ZE//UoXWwYFeytTgKEnMQ== 2024-09-13T09:23:32.865 INFO:teuthology.orchestra.run.smithi190.stdout:[client.0] 2024-09-13T09:23:32.865 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCIBORm38vMCRAA2TpXSrnu6ES9/M28f4x8sQ== 2024-09-13T09:23:32.865 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror] 2024-09-13T09:23:32.865 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCIBORmw34MDRAA4YPerdU4leQ6Z4QTJIluCQ== 2024-09-13T09:23:32.865 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.0] 2024-09-13T09:23:32.865 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCIBORmXmeWEhAAZJQ6L4yvwRJb0iM7Kyd9eg== 2024-09-13T09:23:32.865 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.1] 2024-09-13T09:23:32.865 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCIBORmm0ccGBAAapMruX7E7ZpnwXtBEaKQWQ== 2024-09-13T09:23:32.865 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.2] 2024-09-13T09:23:32.865 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCIBORmrqaoHRAAUTgFR+fpiUkwRo9vM/mv5A== 2024-09-13T09:23:32.865 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.3] 2024-09-13T09:23:32.865 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCIBORm0xkwIxAAf5Yh48EKRdkjLLg6+AbXRA== 2024-09-13T09:23:32.866 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.4] 2024-09-13T09:23:32.866 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCIBORmnVe7KBAA+iU73PHmKcfB2XSaPPPP+g== 2024-09-13T09:23:32.866 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.5] 2024-09-13T09:23:32.866 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCIBORmQHxMLhAAkhsfheLLbPlj548VTCZwuQ== 2024-09-13T09:23:32.866 INFO:teuthology.orchestra.run.smithi190.stdout:[client.mirror.6] 2024-09-13T09:23:32.866 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCIBORmZxrbMxAAS2/pl2X7W1BLc0b8BLaZ1Q== 2024-09-13T09:23:32.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T09:23:32.957 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T09:23:33.009 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T09:23:33.060 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T09:23:33.110 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T09:23:33.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T09:23:33.216 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T09:23:33.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T09:23:33.321 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T09:23:33.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T09:23:33.427 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T09:23:33.480 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T09:23:33.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T09:23:33.586 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-13T09:23:33.587 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-09-13T09:23:33.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-09-13T09:23:33.745 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-09-13T09:23:33.771 DEBUG:teuthology.orchestra.run.smithi190:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-09-13T09:23:33.826 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-09-13T09:23:33.827 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-09-13T09:23:33.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-09-13T09:23:33.868 INFO:tasks.ceph.cluster2.mon.a:Started 2024-09-13T09:23:33.869 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-09-13T09:23:33.869 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-09-13T09:23:33.869 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-09-13T09:23:33.872 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-09-13T09:23:33.872 INFO:tasks.ceph:Setting crush tunables to default 2024-09-13T09:23:33.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster cluster2 osd crush tunables default 2024-09-13T09:23:34.039 INFO:teuthology.orchestra.run.smithi190.stderr:adjusted tunables profile to default 2024-09-13T09:23:34.049 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:34.048+0000 7f9e8e00d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T09:23:34.050 INFO:tasks.ceph:check_enable_crimson: False 2024-09-13T09:23:34.050 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-09-13T09:23:34.051 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:23:34.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-09-13T09:23:34.075 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:23:34.075 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-09-13T09:23:34.091 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:34.090+0000 7f9e8e00d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T09:23:34.141 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:23:34.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-09-13T09:23:34.205 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster cluster2 osd new 49de89ee-dc0d-4ac6-a600-1530e8c29e9d 0 2024-09-13T09:23:34.218 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:34.217+0000 7f9e8e00d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T09:23:34.265 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:34.264+0000 7f9e8e00d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T09:23:34.369 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:34.368+0000 7f9e8e00d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T09:23:34.416 INFO:teuthology.orchestra.run.smithi190.stdout:0 2024-09-13T09:23:34.427 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster cluster2 osd new 143f28ae-3260-47f8-a94b-de76b062584d 1 2024-09-13T09:23:34.460 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:34.459+0000 7f9e8e00d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T09:23:34.596 INFO:teuthology.orchestra.run.smithi190.stdout:1 2024-09-13T09:23:34.607 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster cluster2 osd new 19b0fabd-7a2b-4b10-a9da-b898d489df1f 2 2024-09-13T09:23:34.714 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:34.713+0000 7f9e8e00d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T09:23:34.757 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:34.755+0000 7f9e8e00d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T09:23:34.777 INFO:teuthology.orchestra.run.smithi190.stdout:2 2024-09-13T09:23:34.788 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-09-13T09:23:34.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-09-13T09:23:34.791 INFO:tasks.ceph.cluster2.osd.0:Started 2024-09-13T09:23:34.791 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-09-13T09:23:34.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-09-13T09:23:34.793 INFO:tasks.ceph.cluster2.osd.1:Started 2024-09-13T09:23:34.793 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-09-13T09:23:34.793 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-09-13T09:23:34.795 INFO:tasks.ceph.cluster2.osd.2:Started 2024-09-13T09:23:34.795 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-13T09:23:34.869 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:34.868+0000 7f9e8e00d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T09:23:34.961 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:34.961 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":5,"fsid":"4884b31c-ac19-4f07-b0a6-5fdd65ee73f4","created":"2024-09-13T09:23:33.956282+0000","modified":"2024-09-13T09:23:34.775774+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T09:23:34.775774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"49de89ee-dc0d-4ac6-a600-1530e8c29e9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"143f28ae-3260-47f8-a94b-de76b062584d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"19b0fabd-7a2b-4b10-a9da-b898d489df1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T09:23:34.972 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-09-13T09:23:34.973 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-13T09:23:35.275 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-13T09:23:35.390 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2024-09-13T09:23:35.387+0000 7f00db822640 -1 Falling back to public interface 2024-09-13T09:23:35.395 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2024-09-13T09:23:35.392+0000 7f3620b02640 -1 Falling back to public interface 2024-09-13T09:23:35.395 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2024-09-13T09:23:35.392+0000 7ff86c40e640 -1 Falling back to public interface 2024-09-13T09:23:35.395 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2024-09-13T09:23:35.392+0000 7f361faab640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6808/0: (98) Address already in use 2024-09-13T09:23:35.397 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2024-09-13T09:23:35.395+0000 7f361faab640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6822/0: (98) Address already in use 2024-09-13T09:23:35.431 INFO:teuthology.misc.health.smithi190.stdout: 2024-09-13T09:23:35.431 INFO:teuthology.misc.health.smithi190.stdout:{"epoch":5,"fsid":"4884b31c-ac19-4f07-b0a6-5fdd65ee73f4","created":"2024-09-13T09:23:33.956282+0000","modified":"2024-09-13T09:23:34.775774+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T09:23:34.775774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"49de89ee-dc0d-4ac6-a600-1530e8c29e9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"143f28ae-3260-47f8-a94b-de76b062584d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"19b0fabd-7a2b-4b10-a9da-b898d489df1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T09:23:35.440 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-13T09:23:35.542 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:35.541+0000 7f9e8e00d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T09:23:35.585 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:35.584+0000 7f9e8e00d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T09:23:35.889 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:35.887+0000 7f9e8e00d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T09:23:35.969 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:35.968+0000 7f9e8e00d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T09:23:36.072 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:36.071+0000 7f9e8e00d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T09:23:36.113 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:36.112+0000 7f9e8e00d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T09:23:36.220 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:36.218+0000 7f9e8e00d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T09:23:36.269 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:36.268+0000 7f9e8e00d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T09:23:36.309 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:36.308+0000 7f9e8e00d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T09:23:36.417 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:36.416+0000 7f9e8e00d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T09:23:36.463 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:36.462+0000 7f9e8e00d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T09:23:36.510 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:36.508+0000 7f9e8e00d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T09:23:36.561 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:36.560+0000 7f9e8e00d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T09:23:36.602 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:36.601+0000 7f9e8e00d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T09:23:36.643 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:36.642+0000 7f9e8e00d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T09:23:37.187 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:37.186+0000 7f9e8e00d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T09:23:38.248 INFO:tasks.ceph.cluster2.mgr.x.smithi122.stderr:2024-09-13T09:23:38.247+0000 7f9e2e5bc640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-13T09:23:38.825 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2024-09-13T09:23:38.823+0000 7f00db822640 -1 osd.0 0 log_to_monitors true 2024-09-13T09:23:38.882 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2024-09-13T09:23:38.880+0000 7f3620b02640 -1 osd.1 0 log_to_monitors true 2024-09-13T09:23:38.895 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2024-09-13T09:23:38.894+0000 7ff86c40e640 -1 osd.2 0 log_to_monitors true 2024-09-13T09:23:41.229 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2024-09-13T09:23:41.227+0000 7f00d720c640 -1 osd.0 0 waiting for initial osdmap 2024-09-13T09:23:41.232 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2024-09-13T09:23:41.230+0000 7ff867cfc640 -1 osd.2 0 waiting for initial osdmap 2024-09-13T09:23:41.244 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2024-09-13T09:23:41.243+0000 7f00cf02d640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T09:23:41.248 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2024-09-13T09:23:41.247+0000 7f361d2a6640 -1 osd.1 0 waiting for initial osdmap 2024-09-13T09:23:41.248 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2024-09-13T09:23:41.247+0000 7ff85fb1d640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T09:23:41.258 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2024-09-13T09:23:41.257+0000 7f36148b0640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T09:23:41.744 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-13T09:23:42.009 INFO:teuthology.misc.health.smithi190.stdout: 2024-09-13T09:23:42.010 INFO:teuthology.misc.health.smithi190.stdout:{"epoch":7,"fsid":"4884b31c-ac19-4f07-b0a6-5fdd65ee73f4","created":"2024-09-13T09:23:33.956282+0000","modified":"2024-09-13T09:23:40.250405+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T09:23:34.775774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"49de89ee-dc0d-4ac6-a600-1530e8c29e9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"143f28ae-3260-47f8-a94b-de76b062584d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"19b0fabd-7a2b-4b10-a9da-b898d489df1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T09:23:42.021 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-13T09:23:48.324 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-13T09:23:48.598 INFO:teuthology.misc.health.smithi190.stdout: 2024-09-13T09:23:48.598 INFO:teuthology.misc.health.smithi190.stdout:{"epoch":12,"fsid":"4884b31c-ac19-4f07-b0a6-5fdd65ee73f4","created":"2024-09-13T09:23:33.956282+0000","modified":"2024-09-13T09:23:46.272801+0000","last_up_change":"2024-09-13T09:23:42.232423+0000","last_in_change":"2024-09-13T09:23:34.775774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T09:23:43.424493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"49de89ee-dc0d-4ac6-a600-1530e8c29e9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":218678704},{"type":"v1","addr":"172.21.15.190:6801","nonce":218678704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":218678704},{"type":"v1","addr":"172.21.15.190:6803","nonce":218678704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":218678704},{"type":"v1","addr":"172.21.15.190:6807","nonce":218678704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":218678704},{"type":"v1","addr":"172.21.15.190:6805","nonce":218678704}]},"public_addr":"172.21.15.190:6801/218678704","cluster_addr":"172.21.15.190:6803/218678704","heartbeat_back_addr":"172.21.15.190:6807/218678704","heartbeat_front_addr":"172.21.15.190:6805/218678704","state":["exists","up"]},{"osd":1,"uuid":"143f28ae-3260-47f8-a94b-de76b062584d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6809","nonce":3841277131},{"type":"v1","addr":"172.21.15.190:6811","nonce":3841277131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6813","nonce":3841277131},{"type":"v1","addr":"172.21.15.190:6815","nonce":3841277131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3841277131},{"type":"v1","addr":"172.21.15.190:6823","nonce":3841277131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6817","nonce":3841277131},{"type":"v1","addr":"172.21.15.190:6819","nonce":3841277131}]},"public_addr":"172.21.15.190:6811/3841277131","cluster_addr":"172.21.15.190:6815/3841277131","heartbeat_back_addr":"172.21.15.190:6823/3841277131","heartbeat_front_addr":"172.21.15.190:6819/3841277131","state":["exists","up"]},{"osd":2,"uuid":"19b0fabd-7a2b-4b10-a9da-b898d489df1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3245106308},{"type":"v1","addr":"172.21.15.190:6810","nonce":3245106308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3245106308},{"type":"v1","addr":"172.21.15.190:6814","nonce":3245106308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":3245106308},{"type":"v1","addr":"172.21.15.190:6822","nonce":3245106308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3245106308},{"type":"v1","addr":"172.21.15.190:6818","nonce":3245106308}]},"public_addr":"172.21.15.190:6810/3245106308","cluster_addr":"172.21.15.190:6814/3245106308","heartbeat_back_addr":"172.21.15.190:6822/3245106308","heartbeat_front_addr":"172.21.15.190:6818/3245106308","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:39.855506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:39.931858+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:39.929265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T09:23:48.610 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-13T09:23:48.610 INFO:tasks.ceph:Creating RBD pool 2024-09-13T09:23:48.610 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-09-13T09:23:49.434 INFO:teuthology.orchestra.run.smithi190.stderr:pool 'rbd' created 2024-09-13T09:23:49.447 DEBUG:teuthology.orchestra.run.smithi190:> rbd --cluster cluster2 pool init rbd 2024-09-13T09:23:52.453 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-09-13T09:23:52.454 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-09-13T09:23:52.454 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-09-13T09:23:52.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-09-13T09:23:52.454 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-13T09:23:52.775 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:52.787 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":5,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1456059172},{"type":"v1","addr":"172.21.15.122:6827","nonce":1456059172}]},"active_addr":"172.21.15.122:6827/1456059172","active_change":"2024-09-13T09:23:37.188619+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":862646322}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":1910821065}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":3275324162}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":1751008397}]}]} 2024-09-13T09:23:52.790 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-09-13T09:23:52.790 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-09-13T09:23:52.790 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-13T09:23:53.056 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:53.057 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":16,"fsid":"4884b31c-ac19-4f07-b0a6-5fdd65ee73f4","created":"2024-09-13T09:23:33.956282+0000","modified":"2024-09-13T09:23:52.440217+0000","last_up_change":"2024-09-13T09:23:42.232423+0000","last_in_change":"2024-09-13T09:23:34.775774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T09:23:43.424493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T09:23:48.894975+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"49de89ee-dc0d-4ac6-a600-1530e8c29e9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":218678704},{"type":"v1","addr":"172.21.15.190:6801","nonce":218678704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":218678704},{"type":"v1","addr":"172.21.15.190:6803","nonce":218678704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":218678704},{"type":"v1","addr":"172.21.15.190:6807","nonce":218678704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":218678704},{"type":"v1","addr":"172.21.15.190:6805","nonce":218678704}]},"public_addr":"172.21.15.190:6801/218678704","cluster_addr":"172.21.15.190:6803/218678704","heartbeat_back_addr":"172.21.15.190:6807/218678704","heartbeat_front_addr":"172.21.15.190:6805/218678704","state":["exists","up"]},{"osd":1,"uuid":"143f28ae-3260-47f8-a94b-de76b062584d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6809","nonce":3841277131},{"type":"v1","addr":"172.21.15.190:6811","nonce":3841277131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6813","nonce":3841277131},{"type":"v1","addr":"172.21.15.190:6815","nonce":3841277131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3841277131},{"type":"v1","addr":"172.21.15.190:6823","nonce":3841277131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6817","nonce":3841277131},{"type":"v1","addr":"172.21.15.190:6819","nonce":3841277131}]},"public_addr":"172.21.15.190:6811/3841277131","cluster_addr":"172.21.15.190:6815/3841277131","heartbeat_back_addr":"172.21.15.190:6823/3841277131","heartbeat_front_addr":"172.21.15.190:6819/3841277131","state":["exists","up"]},{"osd":2,"uuid":"19b0fabd-7a2b-4b10-a9da-b898d489df1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3245106308},{"type":"v1","addr":"172.21.15.190:6810","nonce":3245106308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3245106308},{"type":"v1","addr":"172.21.15.190:6814","nonce":3245106308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":3245106308},{"type":"v1","addr":"172.21.15.190:6822","nonce":3245106308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3245106308},{"type":"v1","addr":"172.21.15.190:6818","nonce":3245106308}]},"public_addr":"172.21.15.190:6810/3245106308","cluster_addr":"172.21.15.190:6814/3245106308","heartbeat_back_addr":"172.21.15.190:6822/3245106308","heartbeat_front_addr":"172.21.15.190:6818/3245106308","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:39.855506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:39.931858+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:39.929265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T09:23:53.069 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-09-13T09:23:53.069 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-13T09:23:53.340 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:53.340 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":16,"fsid":"4884b31c-ac19-4f07-b0a6-5fdd65ee73f4","created":"2024-09-13T09:23:33.956282+0000","modified":"2024-09-13T09:23:52.440217+0000","last_up_change":"2024-09-13T09:23:42.232423+0000","last_in_change":"2024-09-13T09:23:34.775774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T09:23:43.424493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T09:23:48.894975+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"49de89ee-dc0d-4ac6-a600-1530e8c29e9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":218678704},{"type":"v1","addr":"172.21.15.190:6801","nonce":218678704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":218678704},{"type":"v1","addr":"172.21.15.190:6803","nonce":218678704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":218678704},{"type":"v1","addr":"172.21.15.190:6807","nonce":218678704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":218678704},{"type":"v1","addr":"172.21.15.190:6805","nonce":218678704}]},"public_addr":"172.21.15.190:6801/218678704","cluster_addr":"172.21.15.190:6803/218678704","heartbeat_back_addr":"172.21.15.190:6807/218678704","heartbeat_front_addr":"172.21.15.190:6805/218678704","state":["exists","up"]},{"osd":1,"uuid":"143f28ae-3260-47f8-a94b-de76b062584d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6809","nonce":3841277131},{"type":"v1","addr":"172.21.15.190:6811","nonce":3841277131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6813","nonce":3841277131},{"type":"v1","addr":"172.21.15.190:6815","nonce":3841277131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3841277131},{"type":"v1","addr":"172.21.15.190:6823","nonce":3841277131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6817","nonce":3841277131},{"type":"v1","addr":"172.21.15.190:6819","nonce":3841277131}]},"public_addr":"172.21.15.190:6811/3841277131","cluster_addr":"172.21.15.190:6815/3841277131","heartbeat_back_addr":"172.21.15.190:6823/3841277131","heartbeat_front_addr":"172.21.15.190:6819/3841277131","state":["exists","up"]},{"osd":2,"uuid":"19b0fabd-7a2b-4b10-a9da-b898d489df1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3245106308},{"type":"v1","addr":"172.21.15.190:6810","nonce":3245106308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3245106308},{"type":"v1","addr":"172.21.15.190:6814","nonce":3245106308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":3245106308},{"type":"v1","addr":"172.21.15.190:6822","nonce":3245106308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3245106308},{"type":"v1","addr":"172.21.15.190:6818","nonce":3245106308}]},"public_addr":"172.21.15.190:6810/3245106308","cluster_addr":"172.21.15.190:6814/3245106308","heartbeat_back_addr":"172.21.15.190:6822/3245106308","heartbeat_front_addr":"172.21.15.190:6818/3245106308","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:39.855506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:39.931858+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:39.929265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T09:23:53.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-09-13T09:23:53.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-09-13T09:23:53.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-09-13T09:23:53.489 INFO:teuthology.orchestra.run.smithi190.stdout:34359738374 2024-09-13T09:23:53.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-13T09:23:53.508 INFO:teuthology.orchestra.run.smithi190.stdout:34359738374 2024-09-13T09:23:53.508 INFO:teuthology.orchestra.run.smithi190.stdout:34359738373 2024-09-13T09:23:53.509 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-13T09:23:53.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-13T09:23:53.773 INFO:teuthology.orchestra.run.smithi190.stdout:34359738373 2024-09-13T09:23:53.785 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2 2024-09-13T09:23:53.832 INFO:teuthology.orchestra.run.smithi190.stdout:34359738372 2024-09-13T09:23:53.837 INFO:teuthology.orchestra.run.smithi190.stdout:34359738373 2024-09-13T09:23:53.846 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.1 2024-09-13T09:23:53.849 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2024-09-13T09:23:54.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-13T09:23:54.847 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-13T09:23:54.849 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-13T09:23:55.077 INFO:teuthology.orchestra.run.smithi190.stdout:34359738373 2024-09-13T09:23:55.090 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2 2024-09-13T09:23:55.135 INFO:teuthology.orchestra.run.smithi190.stdout:34359738372 2024-09-13T09:23:55.148 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.1 2024-09-13T09:23:55.173 INFO:teuthology.orchestra.run.smithi190.stdout:34359738373 2024-09-13T09:23:55.185 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2024-09-13T09:23:56.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-13T09:23:56.149 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-13T09:23:56.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-13T09:23:56.387 INFO:teuthology.orchestra.run.smithi190.stdout:34359738374 2024-09-13T09:23:56.400 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.2 2024-09-13T09:23:56.400 DEBUG:teuthology.parallel:result is None 2024-09-13T09:23:56.444 INFO:teuthology.orchestra.run.smithi190.stdout:34359738373 2024-09-13T09:23:56.457 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.1 2024-09-13T09:23:56.457 DEBUG:teuthology.parallel:result is None 2024-09-13T09:23:56.483 INFO:teuthology.orchestra.run.smithi190.stdout:34359738374 2024-09-13T09:23:56.496 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.0 2024-09-13T09:23:56.496 DEBUG:teuthology.parallel:result is None 2024-09-13T09:23:56.496 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-13T09:23:56.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-13T09:23:56.803 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:56.804 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2024-09-13T09:23:56.816 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-13T09:23:55.195398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81616,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033904,"statfs":{"total":289910292480,"available":289826717696,"internally_reserved":0,"allocated":548864,"data_stored":1303054,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.763014"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:52.448485+0000","last_change":"2024-09-13T09:23:52.448692+0000","last_active":"2024-09-13T09:23:52.448485+0000","last_peered":"2024-09-13T09:23:52.448485+0000","last_clean":"2024-09-13T09:23:52.448485+0000","last_became_active":"2024-09-13T09:23:50.442738+0000","last_became_peered":"2024-09-13T09:23:50.442738+0000","last_unstale":"2024-09-13T09:23:52.448485+0000","last_undegraded":"2024-09-13T09:23:52.448485+0000","last_fullsized":"2024-09-13T09:23:52.448485+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:00:17.513367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072094900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:52.448549+0000","last_change":"2024-09-13T09:23:52.448736+0000","last_active":"2024-09-13T09:23:52.448549+0000","last_peered":"2024-09-13T09:23:52.448549+0000","last_clean":"2024-09-13T09:23:52.448549+0000","last_became_active":"2024-09-13T09:23:50.443108+0000","last_became_peered":"2024-09-13T09:23:50.443108+0000","last_unstale":"2024-09-13T09:23:52.448549+0000","last_undegraded":"2024-09-13T09:23:52.448549+0000","last_fullsized":"2024-09-13T09:23:52.448549+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:24.324564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064757699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:52.448284+0000","last_change":"2024-09-13T09:23:52.448503+0000","last_active":"2024-09-13T09:23:52.448284+0000","last_peered":"2024-09-13T09:23:52.448284+0000","last_clean":"2024-09-13T09:23:52.448284+0000","last_became_active":"2024-09-13T09:23:50.439969+0000","last_became_peered":"2024-09-13T09:23:50.439969+0000","last_unstale":"2024-09-13T09:23:52.448284+0000","last_undegraded":"2024-09-13T09:23:52.448284+0000","last_fullsized":"2024-09-13T09:23:52.448284+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:50.713752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052999299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:52.448742+0000","last_change":"2024-09-13T09:23:52.448963+0000","last_active":"2024-09-13T09:23:52.448742+0000","last_peered":"2024-09-13T09:23:52.448742+0000","last_clean":"2024-09-13T09:23:52.448742+0000","last_became_active":"2024-09-13T09:23:50.442812+0000","last_became_peered":"2024-09-13T09:23:50.442812+0000","last_unstale":"2024-09-13T09:23:52.448742+0000","last_undegraded":"2024-09-13T09:23:52.448742+0000","last_fullsized":"2024-09-13T09:23:52.448742+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:43.338946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088972200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:52.446617+0000","last_change":"2024-09-13T09:23:52.446617+0000","last_active":"2024-09-13T09:23:52.446617+0000","last_peered":"2024-09-13T09:23:52.446617+0000","last_clean":"2024-09-13T09:23:52.446617+0000","last_became_active":"2024-09-13T09:23:50.442111+0000","last_became_peered":"2024-09-13T09:23:50.442111+0000","last_unstale":"2024-09-13T09:23:52.446617+0000","last_undegraded":"2024-09-13T09:23:52.446617+0000","last_fullsized":"2024-09-13T09:23:52.446617+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:15.440943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000891642,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:52.443713+0000","last_change":"2024-09-13T09:23:52.443957+0000","last_active":"2024-09-13T09:23:52.443713+0000","last_peered":"2024-09-13T09:23:52.443713+0000","last_clean":"2024-09-13T09:23:52.443713+0000","last_became_active":"2024-09-13T09:23:50.441154+0000","last_became_peered":"2024-09-13T09:23:50.441154+0000","last_unstale":"2024-09-13T09:23:52.443713+0000","last_undegraded":"2024-09-13T09:23:52.443713+0000","last_fullsized":"2024-09-13T09:23:52.443713+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:40.913446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033978799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:52.443764+0000","last_change":"2024-09-13T09:23:52.444000+0000","last_active":"2024-09-13T09:23:52.443764+0000","last_peered":"2024-09-13T09:23:52.443764+0000","last_clean":"2024-09-13T09:23:52.443764+0000","last_became_active":"2024-09-13T09:23:50.441634+0000","last_became_peered":"2024-09-13T09:23:50.441634+0000","last_unstale":"2024-09-13T09:23:52.443764+0000","last_undegraded":"2024-09-13T09:23:52.443764+0000","last_fullsized":"2024-09-13T09:23:52.443764+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:10.844086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038734700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:52.448242+0000","last_change":"2024-09-13T09:23:52.448440+0000","last_active":"2024-09-13T09:23:52.448242+0000","last_peered":"2024-09-13T09:23:52.448242+0000","last_clean":"2024-09-13T09:23:52.448242+0000","last_became_active":"2024-09-13T09:23:50.440112+0000","last_became_peered":"2024-09-13T09:23:50.440112+0000","last_unstale":"2024-09-13T09:23:52.448242+0000","last_undegraded":"2024-09-13T09:23:52.448242+0000","last_fullsized":"2024-09-13T09:23:52.448242+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:40.210218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048773600000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'97","reported_seq":141,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-13T09:23:51.446195+0000","last_change":"2024-09-13T09:23:45.887443+0000","last_active":"2024-09-13T09:23:51.446195+0000","last_peered":"2024-09-13T09:23:51.446195+0000","last_clean":"2024-09-13T09:23:51.446195+0000","last_became_active":"2024-09-13T09:23:45.886500+0000","last_became_peered":"2024-09-13T09:23:45.886500+0000","last_unstale":"2024-09-13T09:23:51.446195+0000","last_undegraded":"2024-09-13T09:23:51.446195+0000","last_fullsized":"2024-09-13T09:23:51.446195+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:44.266705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:44.266705+0000","last_clean_scrub_stamp":"2024-09-13T09:23:44.266705+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:28.855304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":40760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":631147,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":631147,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T09:23:56.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-13T09:23:57.084 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:57.085 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2024-09-13T09:23:57.096 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-13T09:23:55.195398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81616,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033904,"statfs":{"total":289910292480,"available":289826717696,"internally_reserved":0,"allocated":548864,"data_stored":1303054,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.763014"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:52.448485+0000","last_change":"2024-09-13T09:23:52.448692+0000","last_active":"2024-09-13T09:23:52.448485+0000","last_peered":"2024-09-13T09:23:52.448485+0000","last_clean":"2024-09-13T09:23:52.448485+0000","last_became_active":"2024-09-13T09:23:50.442738+0000","last_became_peered":"2024-09-13T09:23:50.442738+0000","last_unstale":"2024-09-13T09:23:52.448485+0000","last_undegraded":"2024-09-13T09:23:52.448485+0000","last_fullsized":"2024-09-13T09:23:52.448485+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:00:17.513367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072094900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:52.448549+0000","last_change":"2024-09-13T09:23:52.448736+0000","last_active":"2024-09-13T09:23:52.448549+0000","last_peered":"2024-09-13T09:23:52.448549+0000","last_clean":"2024-09-13T09:23:52.448549+0000","last_became_active":"2024-09-13T09:23:50.443108+0000","last_became_peered":"2024-09-13T09:23:50.443108+0000","last_unstale":"2024-09-13T09:23:52.448549+0000","last_undegraded":"2024-09-13T09:23:52.448549+0000","last_fullsized":"2024-09-13T09:23:52.448549+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:24.324564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064757699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:52.448284+0000","last_change":"2024-09-13T09:23:52.448503+0000","last_active":"2024-09-13T09:23:52.448284+0000","last_peered":"2024-09-13T09:23:52.448284+0000","last_clean":"2024-09-13T09:23:52.448284+0000","last_became_active":"2024-09-13T09:23:50.439969+0000","last_became_peered":"2024-09-13T09:23:50.439969+0000","last_unstale":"2024-09-13T09:23:52.448284+0000","last_undegraded":"2024-09-13T09:23:52.448284+0000","last_fullsized":"2024-09-13T09:23:52.448284+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:50.713752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052999299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:52.448742+0000","last_change":"2024-09-13T09:23:52.448963+0000","last_active":"2024-09-13T09:23:52.448742+0000","last_peered":"2024-09-13T09:23:52.448742+0000","last_clean":"2024-09-13T09:23:52.448742+0000","last_became_active":"2024-09-13T09:23:50.442812+0000","last_became_peered":"2024-09-13T09:23:50.442812+0000","last_unstale":"2024-09-13T09:23:52.448742+0000","last_undegraded":"2024-09-13T09:23:52.448742+0000","last_fullsized":"2024-09-13T09:23:52.448742+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:43.338946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088972200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:52.446617+0000","last_change":"2024-09-13T09:23:52.446617+0000","last_active":"2024-09-13T09:23:52.446617+0000","last_peered":"2024-09-13T09:23:52.446617+0000","last_clean":"2024-09-13T09:23:52.446617+0000","last_became_active":"2024-09-13T09:23:50.442111+0000","last_became_peered":"2024-09-13T09:23:50.442111+0000","last_unstale":"2024-09-13T09:23:52.446617+0000","last_undegraded":"2024-09-13T09:23:52.446617+0000","last_fullsized":"2024-09-13T09:23:52.446617+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:15.440943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000891642,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:52.443713+0000","last_change":"2024-09-13T09:23:52.443957+0000","last_active":"2024-09-13T09:23:52.443713+0000","last_peered":"2024-09-13T09:23:52.443713+0000","last_clean":"2024-09-13T09:23:52.443713+0000","last_became_active":"2024-09-13T09:23:50.441154+0000","last_became_peered":"2024-09-13T09:23:50.441154+0000","last_unstale":"2024-09-13T09:23:52.443713+0000","last_undegraded":"2024-09-13T09:23:52.443713+0000","last_fullsized":"2024-09-13T09:23:52.443713+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:40.913446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033978799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:52.443764+0000","last_change":"2024-09-13T09:23:52.444000+0000","last_active":"2024-09-13T09:23:52.443764+0000","last_peered":"2024-09-13T09:23:52.443764+0000","last_clean":"2024-09-13T09:23:52.443764+0000","last_became_active":"2024-09-13T09:23:50.441634+0000","last_became_peered":"2024-09-13T09:23:50.441634+0000","last_unstale":"2024-09-13T09:23:52.443764+0000","last_undegraded":"2024-09-13T09:23:52.443764+0000","last_fullsized":"2024-09-13T09:23:52.443764+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:10.844086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038734700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T09:23:52.448242+0000","last_change":"2024-09-13T09:23:52.448440+0000","last_active":"2024-09-13T09:23:52.448242+0000","last_peered":"2024-09-13T09:23:52.448242+0000","last_clean":"2024-09-13T09:23:52.448242+0000","last_became_active":"2024-09-13T09:23:50.440112+0000","last_became_peered":"2024-09-13T09:23:50.440112+0000","last_unstale":"2024-09-13T09:23:52.448242+0000","last_undegraded":"2024-09-13T09:23:52.448242+0000","last_fullsized":"2024-09-13T09:23:52.448242+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:40.210218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048773600000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'97","reported_seq":141,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-13T09:23:51.446195+0000","last_change":"2024-09-13T09:23:45.887443+0000","last_active":"2024-09-13T09:23:51.446195+0000","last_peered":"2024-09-13T09:23:51.446195+0000","last_clean":"2024-09-13T09:23:51.446195+0000","last_became_active":"2024-09-13T09:23:45.886500+0000","last_became_peered":"2024-09-13T09:23:45.886500+0000","last_unstale":"2024-09-13T09:23:51.446195+0000","last_undegraded":"2024-09-13T09:23:51.446195+0000","last_fullsized":"2024-09-13T09:23:51.446195+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:44.266705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:44.266705+0000","last_clean_scrub_stamp":"2024-09-13T09:23:44.266705+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:28.855304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":40760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":631147,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":631147,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T09:23:57.097 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-13T09:23:57.097 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-09-13T09:23:57.097 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-09-13T09:23:57.098 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-09-13T09:23:57.402 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T09:23:57.402 INFO:teuthology.orchestra.run.smithi190.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T09:23:57.415 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-09-13T09:23:57.415 INFO:teuthology.run_tasks:Running task exec... 2024-09-13T09:23:57.425 INFO:teuthology.task.exec:Executing custom commands... 2024-09-13T09:23:57.425 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi190.front.sepia.ceph.com 2024-09-13T09:23:57.425 DEBUG:teuthology.orchestra.run.smithi190:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-13T09:23:57.765 INFO:teuthology.orchestra.run.smithi190.stderr:updated caps for client.mirror 2024-09-13T09:23:57.779 DEBUG:teuthology.orchestra.run.smithi190:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-13T09:23:58.119 INFO:teuthology.orchestra.run.smithi190.stderr:updated caps for client.mirror.0 2024-09-13T09:23:58.132 DEBUG:teuthology.orchestra.run.smithi190:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-13T09:23:58.476 INFO:teuthology.orchestra.run.smithi190.stderr:updated caps for client.mirror.1 2024-09-13T09:23:58.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-13T09:23:58.833 INFO:teuthology.orchestra.run.smithi190.stderr:updated caps for client.mirror.2 2024-09-13T09:23:58.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-13T09:23:59.187 INFO:teuthology.orchestra.run.smithi190.stderr:updated caps for client.mirror.3 2024-09-13T09:23:59.201 DEBUG:teuthology.orchestra.run.smithi190:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-13T09:23:59.545 INFO:teuthology.orchestra.run.smithi190.stderr:updated caps for client.mirror 2024-09-13T09:23:59.559 DEBUG:teuthology.orchestra.run.smithi190:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-13T09:23:59.904 INFO:teuthology.orchestra.run.smithi190.stderr:updated caps for client.mirror.0 2024-09-13T09:23:59.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-13T09:24:00.262 INFO:teuthology.orchestra.run.smithi190.stderr:updated caps for client.mirror.1 2024-09-13T09:24:00.275 DEBUG:teuthology.orchestra.run.smithi190:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-13T09:24:00.616 INFO:teuthology.orchestra.run.smithi190.stderr:updated caps for client.mirror.2 2024-09-13T09:24:00.630 DEBUG:teuthology.orchestra.run.smithi190:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-13T09:24:00.974 INFO:teuthology.orchestra.run.smithi190.stderr:updated caps for client.mirror.3 2024-09-13T09:24:00.988 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-13T09:24:00.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-13T09:24:00.999 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-13T09:24:01.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-13T09:24:01.001 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-13T09:24:01.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-13T09:24:01.040 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-13T09:24:01.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-13T09:24:01.043 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-13T09:24:01.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-13T09:24:01.075 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-13T09:24:01.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-13T09:24:01.077 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-13T09:24:01.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-13T09:24:01.107 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-13T09:24:01.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-13T09:24:01.110 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-13T09:24:01.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-13T09:24:01.141 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-13T09:24:01.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-13T09:24:01.143 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-13T09:24:01.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-13T09:24:01.174 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-13T09:24:01.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-13T09:24:01.177 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-13T09:24:01.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-13T09:24:01.208 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-13T09:24:01.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-13T09:24:01.210 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-13T09:24:01.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-13T09:24:01.241 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-13T09:24:01.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-13T09:24:01.243 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-13T09:24:01.276 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726219441 2024-09-13T09:24:01.277 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-13T09:24:01.277 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-13T09:24:01.309 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726219441 2024-09-13T09:24:01.310 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-13T09:24:01.310 INFO:teuthology.run_tasks:Running task workunit... 2024-09-13T09:24:01.346 INFO:tasks.workunit:Pulling workunits from ref 7a853b1706d076d58e0da7a5d3b9b9eb36276880 2024-09-13T09:24:01.347 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-13T09:24:01.347 DEBUG:teuthology.orchestra.run.smithi190:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-13T09:24:01.348 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2024-09-13T09:24:01.348 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2024-09-13T09:24:01.348 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2024-09-13T09:24:01.348 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2024-09-13T09:24:01.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T09:24:01.363 INFO:teuthology.orchestra.run.smithi190.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-09-13T09:24:01.363 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-13T09:24:01.418 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-13T09:24:01.419 DEBUG:teuthology.orchestra.run.smithi190:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-09-13T09:24:01.473 INFO:tasks.workunit:timeout=6h 2024-09-13T09:24:01.473 INFO:tasks.workunit:cleanup=True 2024-09-13T09:24:01.474 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 7a853b1706d076d58e0da7a5d3b9b9eb36276880 2024-09-13T09:24:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-09-13T09:24:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Note: switching to '7a853b1706d076d58e0da7a5d3b9b9eb36276880'. 2024-09-13T09:24:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:24:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-13T09:24:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-13T09:24:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:state without impacting any branches by switching back to a branch. 2024-09-13T09:24:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:24:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-13T09:24:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-13T09:24:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:24:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: git switch -c 2024-09-13T09:24:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:24:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Or undo this operation with: 2024-09-13T09:24:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:24:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: git switch - 2024-09-13T09:24:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:24:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-13T09:24:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:24:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:HEAD is now at 7a853b1706d Merge pull request #59683 from zdover23/wip-doc-2024-09-10-backport-59677-to-reef 2024-09-13T09:24:57.447 DEBUG:teuthology.orchestra.run.smithi190:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-09-13T09:24:57.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-13T09:24:57.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-13T09:24:57.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-13T09:24:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-13T09:24:58.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-13T09:24:58.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-13T09:24:58.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-13T09:24:58.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-13T09:24:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-13T09:24:58.494 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-13T09:24:58.494 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-09-13T09:24:58.510 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-09-13T09:24:58.511 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-09-13T09:24:58.511 DEBUG:teuthology.orchestra.run.smithi190:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7a853b1706d076d58e0da7a5d3b9b9eb36276880 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-13T09:24:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ IMAGE_COUNT=50 2024-09-13T09:24:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ export LOCKDEP=0 2024-09-13T09:24:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ LOCKDEP=0 2024-09-13T09:24:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-13T09:24:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-09-13T09:24:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ type xmlstarlet 2024-09-13T09:24:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ XMLSTARLET=xmlstarlet 2024-09-13T09:24:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ RBD_MIRROR_INSTANCES=4 2024-09-13T09:24:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CLUSTER1=cluster1 2024-09-13T09:24:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CLUSTER2=cluster2 2024-09-13T09:24:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ PEER_CLUSTER_SUFFIX= 2024-09-13T09:24:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ POOL=mirror 2024-09-13T09:24:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ PARENT_POOL=mirror_parent 2024-09-13T09:24:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ NS1=ns1 2024-09-13T09:24:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ NS2=ns2 2024-09-13T09:24:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ TEMPDIR= 2024-09-13T09:24:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ID=mirror 2024-09-13T09:24:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock 2024-09-13T09:24:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-09-13T09:24:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ MIRROR_POOL_MODE=image 2024-09-13T09:24:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ MIRROR_IMAGE_MODE=snapshot 2024-09-13T09:24:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-09-13T09:24:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS='--id mirror' 2024-09-13T09:24:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ LAST_MIRROR_INSTANCE=3 2024-09-13T09:24:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-13T09:24:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-09-13T09:24:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-09-13T09:24:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_BIN=. 2024-09-13T09:24:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_SRC=. 2024-09-13T09:24:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' -e CMakeCache.txt ']' 2024-09-13T09:24:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 -gt 0 ']' 2024-09-13T09:24:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ setup 2024-09-13T09:24:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local c 2024-09-13T09:24:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-09-13T09:24:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ setup_tempdir 2024-09-13T09:24:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:24:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mktemp -d 2024-09-13T09:24:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ TEMPDIR=/tmp/tmp.V7YewHEkvl 2024-09-13T09:24:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -z 1 ']' 2024-09-13T09:24:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ setup_pools cluster1 cluster2 2024-09-13T09:24:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:24:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local remote_cluster=cluster2 2024-09-13T09:24:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mon_map_file 2024-09-13T09:24:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mon_addr 2024-09-13T09:24:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local admin_key_file 2024-09-13T09:24:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local uuid 2024-09-13T09:24:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= 2024-09-13T09:24:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-09-13T09:24:59.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:pool 'mirror' already exists 2024-09-13T09:24:59.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= 2024-09-13T09:24:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-09-13T09:25:00.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:pool 'mirror_parent' already exists 2024-09-13T09:25:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= 2024-09-13T09:25:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 pool init mirror 2024-09-13T09:25:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= 2024-09-13T09:25:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-09-13T09:25:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:25:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= 2024-09-13T09:25:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-09-13T09:25:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-09-13T09:25:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-09-13T09:25:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-09-13T09:25:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-09-13T09:25:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-09-13T09:25:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -z ']' 2024-09-13T09:25:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -z ']' 2024-09-13T09:25:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ peer_add cluster1 mirror cluster2 2024-09-13T09:25:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:25:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:25:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:25:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:25:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local client_cluster=cluster2 2024-09-13T09:25:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:25:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local remote_cluster=cluster2 2024-09-13T09:25:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local uuid_var_name 2024-09-13T09:25:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:25:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local error_code 2024-09-13T09:25:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local peer_uuid 2024-09-13T09:25:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 16 32 2024-09-13T09:25:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:25:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-09-13T09:25:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ peer_uuid=2d6488a4-24bf-4786-9ac9-30ee9f992f4e 2024-09-13T09:25:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=0 2024-09-13T09:25:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:25:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 0 -eq 17 ']' 2024-09-13T09:25:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 -eq 0 2024-09-13T09:25:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:25:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:25:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-09-13T09:25:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:25:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:25:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror_parent 2024-09-13T09:25:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:25:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local client_cluster=cluster2 2024-09-13T09:25:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:25:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local remote_cluster=cluster2 2024-09-13T09:25:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local uuid_var_name 2024-09-13T09:25:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:25:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local error_code 2024-09-13T09:25:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local peer_uuid 2024-09-13T09:25:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 16 32 2024-09-13T09:25:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:25:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-09-13T09:25:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ peer_uuid=9c373768-63a2-4728-909c-e5d3e36cb6c7 2024-09-13T09:25:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=0 2024-09-13T09:25:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:25:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 0 -eq 17 ']' 2024-09-13T09:25:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 -eq 0 2024-09-13T09:25:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:25:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:25:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ setup_pools cluster2 cluster1 2024-09-13T09:25:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:25:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local remote_cluster=cluster1 2024-09-13T09:25:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mon_map_file 2024-09-13T09:25:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mon_addr 2024-09-13T09:25:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local admin_key_file 2024-09-13T09:25:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local uuid 2024-09-13T09:25:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= 2024-09-13T09:25:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-09-13T09:25:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:pool 'mirror' already exists 2024-09-13T09:25:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= 2024-09-13T09:25:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-09-13T09:25:08.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:pool 'mirror_parent' already exists 2024-09-13T09:25:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= 2024-09-13T09:25:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 pool init mirror 2024-09-13T09:25:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= 2024-09-13T09:25:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-09-13T09:25:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:25:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= 2024-09-13T09:25:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-09-13T09:25:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-09-13T09:25:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-09-13T09:25:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-09-13T09:25:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-09-13T09:25:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-09-13T09:25:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -z ']' 2024-09-13T09:25:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -z ']' 2024-09-13T09:25:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ peer_add cluster2 mirror cluster1 2024-09-13T09:25:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:25:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:25:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:25:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:25:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local client_cluster=cluster1 2024-09-13T09:25:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:25:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local remote_cluster=cluster1 2024-09-13T09:25:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local uuid_var_name 2024-09-13T09:25:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:25:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local error_code 2024-09-13T09:25:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local peer_uuid 2024-09-13T09:25:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 16 32 2024-09-13T09:25:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:25:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-09-13T09:25:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ peer_uuid=7a7dd7ef-01a2-40fd-83db-5f0134bd679f 2024-09-13T09:25:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=0 2024-09-13T09:25:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:25:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 0 -eq 17 ']' 2024-09-13T09:25:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 -eq 0 2024-09-13T09:25:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:25:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:25:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-09-13T09:25:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:25:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:25:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror_parent 2024-09-13T09:25:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:25:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local client_cluster=cluster1 2024-09-13T09:25:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:25:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local remote_cluster=cluster1 2024-09-13T09:25:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local uuid_var_name 2024-09-13T09:25:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:25:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local error_code 2024-09-13T09:25:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local peer_uuid 2024-09-13T09:25:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 16 32 2024-09-13T09:25:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:25:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-09-13T09:25:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ peer_uuid=25e58861-347a-4a47-a9c4-e461fae0678b 2024-09-13T09:25:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=0 2024-09-13T09:25:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:25:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 0 -eq 17 ']' 2024-09-13T09:25:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 -eq 0 2024-09-13T09:25:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:25:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:25:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:25:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirrors cluster1 2024-09-13T09:25:14.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:25:14.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ seq 0 3 2024-09-13T09:25:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirror cluster1:0 2024-09-13T09:25:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2024-09-13T09:25:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2024-09-13T09:25:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:25:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:0 2024-09-13T09:25:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2024-09-13T09:25:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2024-09-13T09:25:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2024-09-13T09:25:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2024-09-13T09:25:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-13T09:25:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2024-09-13T09:25:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2024-09-13T09:25:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:25:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2024-09-13T09:25:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2024-09-13T09:25:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirror cluster1:1 2024-09-13T09:25:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2024-09-13T09:25:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2024-09-13T09:25:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:25:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:1 2024-09-13T09:25:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2024-09-13T09:25:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2024-09-13T09:25:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2024-09-13T09:25:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2024-09-13T09:25:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-13T09:25:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2024-09-13T09:25:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2024-09-13T09:25:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:25:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2024-09-13T09:25:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2024-09-13T09:25:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirror cluster1:2 2024-09-13T09:25:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2024-09-13T09:25:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2024-09-13T09:25:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:25:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:2 2024-09-13T09:25:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2024-09-13T09:25:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2024-09-13T09:25:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2024-09-13T09:25:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2024-09-13T09:25:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 2 = cluster1:2 ']' 2024-09-13T09:25:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2024-09-13T09:25:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2024-09-13T09:25:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:25:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2024-09-13T09:25:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2024-09-13T09:25:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirror cluster1:3 2024-09-13T09:25:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2024-09-13T09:25:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2024-09-13T09:25:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:25:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1:3 2024-09-13T09:25:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2024-09-13T09:25:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2024-09-13T09:25:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2024-09-13T09:25:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2024-09-13T09:25:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 3 = cluster1:3 ']' 2024-09-13T09:25:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2024-09-13T09:25:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2024-09-13T09:25:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:25:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2024-09-13T09:25:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2024-09-13T09:25:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirrors cluster2 2024-09-13T09:25:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:25:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ seq 0 3 2024-09-13T09:25:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirror cluster2:0 2024-09-13T09:25:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2:0 2024-09-13T09:25:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2024-09-13T09:25:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-13T09:25:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster2:0 2024-09-13T09:25:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2024-09-13T09:25:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2024-09-13T09:25:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster2 2024-09-13T09:25:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2024-09-13T09:25:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-13T09:25:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster2 2024-09-13T09:25:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster2 2024-09-13T09:25:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2024-09-13T09:25:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:25:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2024-09-13T09:25:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2024-09-13T09:25:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirror cluster2:1 2024-09-13T09:25:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2:1 2024-09-13T09:25:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2024-09-13T09:25:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-13T09:25:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster2:1 2024-09-13T09:25:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2024-09-13T09:25:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2024-09-13T09:25:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster2 2024-09-13T09:25:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=1 2024-09-13T09:25:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-13T09:25:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster2 2024-09-13T09:25:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster2 2024-09-13T09:25:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=1 2024-09-13T09:25:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:25:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2024-09-13T09:25:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2024-09-13T09:25:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirror cluster2:2 2024-09-13T09:25:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2:2 2024-09-13T09:25:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2024-09-13T09:25:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster2:2 cluster instance 2024-09-13T09:25:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster2:2 2024-09-13T09:25:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2024-09-13T09:25:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2024-09-13T09:25:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster2 2024-09-13T09:25:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=2 2024-09-13T09:25:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 2 = cluster2:2 ']' 2024-09-13T09:25:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster2 2024-09-13T09:25:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster2 2024-09-13T09:25:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=2 2024-09-13T09:25:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:25:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2024-09-13T09:25:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2024-09-13T09:25:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirror cluster2:3 2024-09-13T09:25:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2:3 2024-09-13T09:25:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2024-09-13T09:25:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster2:3 cluster instance 2024-09-13T09:25:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster2:3 2024-09-13T09:25:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2024-09-13T09:25:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2024-09-13T09:25:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster2 2024-09-13T09:25:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=3 2024-09-13T09:25:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 3 = cluster2:3 ']' 2024-09-13T09:25:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster2 2024-09-13T09:25:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster2 2024-09-13T09:25:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=3 2024-09-13T09:25:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:25:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2024-09-13T09:25:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2024-09-13T09:25:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-09-13T09:25:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee -a /tmp/tmp.V7YewHEkvl/rbd-mirror.test.log 2024-09-13T09:25:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ date '+%F %T' 2024-09-13T09:25:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo 2024-09-13 09:25:14 TEST: add image and test replay after client crashes 2024-09-13T09:25:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:2024-09-13 09:25:14 TEST: add image and test replay after client crashes 2024-09-13T09:25:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=test 2024-09-13T09:25:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2024-09-13T09:25:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:25:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:25:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:25:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:25:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:25:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:25:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:25:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:25:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:25:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror test 512M 2024-09-13T09:25:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:25:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:25:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:25:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:25:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:25:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:25:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:25:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 512M ']' 2024-09-13T09:25:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2024-09-13T09:25:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:25:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/test 2024-09-13T09:25:14.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:25:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror test snapshot 2024-09-13T09:25:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:25:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:25:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:25:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:25:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2024-09-13T09:25:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:25:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/test 2024-09-13T09:25:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2024-09-13T09:25:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T09:25:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:25:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:25:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10abb2f0df40 2024-09-13T09:25:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10abb2f0df40 2024-09-13T09:25:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:25:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:25:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:25:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:25:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:25:14 2024 2024-09-13T09:25:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:25:14 2024 2024-09-13T09:25:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:25:14 2024 2024-09-13T09:25:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:25:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:25:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:25:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:25:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-13T09:25:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:25:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:25:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:25:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-13T09:25:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:25:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:25:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:25:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:25:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:25:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:25:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:25:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-13T09:25:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:25:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:25:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:25:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:25:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:25:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:25:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-13T09:25:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:25:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:25:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:25:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:25:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:25:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:25:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:25:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:25:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:25:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:25:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:25:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:25:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:25:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:25:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:25:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:25:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:25:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:25:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:25:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:25:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:25:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:25:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:25:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:25:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:25:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:25:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:25:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:25:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:25:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:25:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:25:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:25:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:25:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:25:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:25:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:25:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:25:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:25:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:25:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:25:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:25:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:25:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:25:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:25:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:25:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:25:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:25:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:25:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:25:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:25:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:25:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:25:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:25:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:25:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:25:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:25:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:25:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:25:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:25:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:25:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:25:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:25:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:25:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:25:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:25:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:25:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:25:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:25:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:25:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:25:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:25:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:25:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:25:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:25:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:25:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:25:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:25:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:25:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:25:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:25:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:25:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:25:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:25:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:25:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:25:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:25:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:25:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:25:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:25:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:25:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:25:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:25:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:25:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:25:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:25:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:25:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:25:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:25:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:25:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:25:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:25:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:25:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:25:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:25:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:25:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:25:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:25:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:25:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:25:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:25:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:25:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:25:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:25:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:25:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:25:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-13T09:25:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:25:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:25:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:25:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:25:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:25:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:25:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:25:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:25:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:25:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:25:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:25:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:25:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:25:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:25:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:25:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:25:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:25:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:25:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:25:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:25:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:25:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:25:16.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:25:16.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:25:16.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:25:16.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:25:16.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:25:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:25:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:25:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:25:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-13T09:25:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:25:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:25:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:25:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:25:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 1 2024-09-13T09:25:17.348 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-13T09:25:17.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-13T09:25:17.350 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-13T09:25:17.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:25:17.349+0000 7fdd7dc32640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 36376) UID: 1000 2024-09-13T09:25:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:25:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:25:17.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:17.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:25:17.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:25:17.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:25:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:25:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:25:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:25:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:25:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:25:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:25:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:25:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:25:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:25:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:25:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:25:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:25:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:25:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:25:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:25:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:25:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:25:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:25:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:25:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:25:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:25:17.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:25:17.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:25:17.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:25:17.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:25:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:25:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:25:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:25:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:25:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:25:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:25:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:25:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:25:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:25:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:25:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:25:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:25:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:25:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:25:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:25:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:25:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:25:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:25:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:25:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:25:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:25:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:25:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:25:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:25:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:25:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:25:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:25:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:25:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:25:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:25:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:25:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:25:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:25:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:25:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:25:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:25:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:25:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:25:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:25:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:25:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:25:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:25:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:25:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:25:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:25:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:25:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:25:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:25:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:25:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:25:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:25:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:25:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:25:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:25:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:25:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:25:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:25:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:25:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:25:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:25:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:25:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:25:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:25:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:25:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:25:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:25:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:25:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:25:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:25:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:25:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:25:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:25:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:25:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:25:18.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:25:18.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:25:18.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:25:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-13T09:25:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:25:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:25:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:25:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:25:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:25:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:25:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:25:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:25:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:25:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:25:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:25:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:25:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:25:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:25:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:25:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:25:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:25:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:25:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:25:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:25:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:25:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:25:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:25:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:25:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:25:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:25:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:25:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:25:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:25:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:25:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-13T09:25:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:25:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:25:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:25:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:25:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 2 2024-09-13T09:25:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:25:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:25:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:25:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:25:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:25:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:25:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:25:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:25:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:25:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:25:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:25:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:25:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:25:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:25:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:25:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:25:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:25:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:25:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:25:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:25:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:25:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:25:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:25:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:25:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:25:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:25:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:25:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:25:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:25:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:25:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:25:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:25:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:25:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:25:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:25:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:25:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:25:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:25:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:25:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:25:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:25:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:25:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:25:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:25:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:25:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:25:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:25:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:25:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:25:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:25:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:25:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:25:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:25:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:25:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:25:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:25:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:25:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:25:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:25:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:25:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:25:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:25:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:25:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:25:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:25:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:25:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:25:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:25:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:25:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:25:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:25:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:25:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:25:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:25:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:25:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:25:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:25:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:25:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:25:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:25:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:25:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:25:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:25:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:25:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:25:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:25:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:25:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:25:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:25:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:25:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:25:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:25:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:25:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:25:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-13T09:25:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:25:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:25:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:25:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:25:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:25:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:25:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:25:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:25:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:25:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:25:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:25:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:25:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:25:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:25:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:25:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:25:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:25:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:25:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:25:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:25:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:25:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:25:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:25:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:25:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:25:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:25:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:25:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:25:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:25:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:25:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-13T09:25:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:25:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:25:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:25:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:25:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 4 2024-09-13T09:25:24.350 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-13T09:25:24.350 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:25:24.351 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-13T09:25:24.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-13T09:25:24.351 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-13T09:25:24.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-13T09:25:24.355 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2024-09-13T09:25:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:25:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:25:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:25:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:25:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:25:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:25:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:25:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:25:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:25:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:25:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:25:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:25:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:25:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:25:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:25:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:25:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:25:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:25:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:25:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:25:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:25:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:25:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:25:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:25:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:25:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:25:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:25:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:25:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:25:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:25:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:25:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:25:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:25:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:25:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:25:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:25:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:25:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:25:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:25:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:25:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:25:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:25:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:25:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:25:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:25:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:25:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:25:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:25:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:25:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:25:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:25:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:25:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:25:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:25:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:25:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:25:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:25:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:25:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:25:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:25:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:25:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:25:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:25:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:25:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:25:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:25:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:25:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:25:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:25:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:25:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:25:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:25:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:25:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:25:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:25:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:25:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:25:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:25:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:25:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:25:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:25:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:25:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:25:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:25:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:25:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:25:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:25:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:25:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:25:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:25:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:25:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:25:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:25:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:25:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:25:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:25:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:25:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:25:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:25:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:25:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:25:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:25:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:25:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:25:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:25:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:25:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:25:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-13T09:25:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:25:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:25:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:25:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:25:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:25:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:25:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:25:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:25:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:25:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:25:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:25:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:25:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:25:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:25:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:25:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:25:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:25:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:25:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:25:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:25:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:25:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:25:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:25:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:25:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:25:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:25:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:25:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:25:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:25:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:25:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-13T09:25:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:25:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:25:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:25:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:25:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 8 2024-09-13T09:25:25.348 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5164324903807587) > weight (0.25) 2024-09-13T09:25:25.348 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7965079091156345) > weight (0.25) 2024-09-13T09:25:25.349 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-13T09:25:25.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-13T09:25:25.349 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-13T09:25:25.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T09:25:25.348+0000 7f0297e4c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 36310) UID: 1000 2024-09-13T09:25:29.350 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-13T09:25:29.351 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:25:29.351 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-13T09:25:29.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-13T09:25:29.351 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-13T09:25:29.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-13T09:25:29.355 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2024-09-13T09:25:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:25:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:25:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:25:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:25:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:25:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:25:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:25:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:25:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:25:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:25:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:25:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:25:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:25:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:25:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:25:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:25:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:25:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:25:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:25:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:25:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:25:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:25:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:25:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:25:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:25:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:25:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:25:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:25:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:25:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:25:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:25:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:25:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:25:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:25:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:25:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:25:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:25:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:25:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:25:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:25:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:25:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:25:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:25:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:25:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:25:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:25:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:25:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:25:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:25:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:25:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:25:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:25:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:25:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:25:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:25:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:25:33.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:25:33.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:25:33.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:33.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:25:33.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:25:33.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:25:33.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:25:33.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:25:33.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:25:33.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:25:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:25:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:25:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:25:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:25:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:25:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:25:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:25:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:25:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:25:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:25:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:25:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:25:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:25:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:25:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:25:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:25:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:25:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:25:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:25:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:25:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:25:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:25:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:25:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:25:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:25:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:25:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:25:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:25:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:25:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:25:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:25:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:25:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:25:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:25:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:25:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:25:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:25:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:25:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:25:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:25:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:25:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:25:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:25:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:25:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:25:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:25:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:25:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:25:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:25:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:25:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name= 2024-09-13T09:25:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ seq 1 10 2024-09-13T09:25:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:25:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror test 2024-09-13T09:25:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:25:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:25:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:25:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:25:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=5 2024-09-13T09:25:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:25:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-13T09:25:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 7874560 bytes 2024-09-13T09:25:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 37500928 bytes 2024-09-13T09:25:33.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 40236032 bytes 2024-09-13T09:25:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 82356224 bytes 2024-09-13T09:25:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 154647552 bytes 2024-09-13T09:25:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 199063552 bytes 2024-09-13T09:25:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 235940864 bytes 2024-09-13T09:25:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 281352192 bytes 2024-09-13T09:25:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 337215488 bytes 2024-09-13T09:25:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 362678272 bytes 2024-09-13T09:25:34.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 402784256 bytes 2024-09-13T09:25:34.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 419514368 bytes 2024-09-13T09:25:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 461438976 bytes 2024-09-13T09:25:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 527159296 bytes 2024-09-13T09:25:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 536239104 bytes 2024-09-13T09:25:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 547216384 bytes 2024-09-13T09:25:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 650378240 bytes 2024-09-13T09:25:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 696931328 bytes 2024-09-13T09:25:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 753306624 bytes 2024-09-13T09:25:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 842573824 bytes 2024-09-13T09:25:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 877777920 bytes 2024-09-13T09:25:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 961940480 bytes 2024-09-13T09:25:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1002079232 bytes 2024-09-13T09:25:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1024278528 bytes 2024-09-13T09:25:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1034336256 bytes 2024-09-13T09:25:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1105241088 bytes 2024-09-13T09:25:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1146044416 bytes 2024-09-13T09:25:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1196704768 bytes 2024-09-13T09:25:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1248247808 bytes 2024-09-13T09:25:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1342253056 bytes 2024-09-13T09:25:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1377105920 bytes 2024-09-13T09:25:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1388440576 bytes 2024-09-13T09:25:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1421051904 bytes 2024-09-13T09:25:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1483999232 bytes 2024-09-13T09:25:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1572036608 bytes 2024-09-13T09:25:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1577085952 bytes 2024-09-13T09:25:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1648591872 bytes 2024-09-13T09:25:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1701873664 bytes 2024-09-13T09:25:36.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1743038464 bytes 2024-09-13T09:25:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1778395136 bytes 2024-09-13T09:25:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1833222144 bytes 2024-09-13T09:25:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1891189760 bytes 2024-09-13T09:25:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1942839296 bytes 2024-09-13T09:25:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1974290432 bytes 2024-09-13T09:25:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2031400960 bytes 2024-09-13T09:25:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2083109888 bytes 2024-09-13T09:25:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2088062976 bytes 2024-09-13T09:25:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2136830976 bytes 2024-09-13T09:25:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2203232256 bytes 2024-09-13T09:25:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2217651200 bytes 2024-09-13T09:25:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2269954048 bytes 2024-09-13T09:25:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2308670464 bytes 2024-09-13T09:25:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:25:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:25:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:25:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:25:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-13T09:25:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:25:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:25:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:25:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:25:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:25:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:25:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:25:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:25:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-13T09:25:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:25:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:25:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:25:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:25:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:25:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:25:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:25:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:25:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-13T09:25:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='test: 2024-09-13T09:25:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:25:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: down+unknown 2024-09-13T09:25:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: status not found 2024-09-13T09:25:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13T09:25:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2024-09-13T09:25:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2024-09-13T09:25:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: down+unknown 2024-09-13T09:25:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: status not found 2024-09-13T09:25:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: ' 2024-09-13T09:25:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ state: .*up+replaying' 2024-09-13T09:25:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 1 2024-09-13T09:25:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:25:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 2 2024-09-13T09:25:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-13T09:25:41.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:25:41.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:25:41.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:25:41.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:25:41.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:25:41.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:25:41.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:25:41.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:25:41.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-13T09:25:41.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='test: 2024-09-13T09:25:41.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:25:41.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: down+unknown 2024-09-13T09:25:41.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: status not found 2024-09-13T09:25:41.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13T09:25:41.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2024-09-13T09:25:41.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2024-09-13T09:25:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: down+unknown 2024-09-13T09:25:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: status not found 2024-09-13T09:25:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: ' 2024-09-13T09:25:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ state: .*up+replaying' 2024-09-13T09:25:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 1 2024-09-13T09:25:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:25:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 4 2024-09-13T09:25:45.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-13T09:25:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:25:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:25:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:25:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:25:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:25:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:25:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:25:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:25:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-13T09:25:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='test: 2024-09-13T09:25:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:25:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: down+unknown 2024-09-13T09:25:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: status not found 2024-09-13T09:25:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13T09:25:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2024-09-13T09:25:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2024-09-13T09:25:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: down+unknown 2024-09-13T09:25:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: status not found 2024-09-13T09:25:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: ' 2024-09-13T09:25:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ state: .*up+replaying' 2024-09-13T09:25:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 1 2024-09-13T09:25:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:25:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 8 2024-09-13T09:25:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-13T09:25:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:25:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:25:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:25:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:25:53.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:25:53.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:25:53.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:25:53.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:25:53.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-13T09:25:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='test: 2024-09-13T09:25:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:25:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: down+unknown 2024-09-13T09:25:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: status not found 2024-09-13T09:25:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13T09:25:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2024-09-13T09:25:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2024-09-13T09:25:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: down+unknown 2024-09-13T09:25:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: status not found 2024-09-13T09:25:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: ' 2024-09-13T09:25:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ state: .*up+replaying' 2024-09-13T09:25:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 1 2024-09-13T09:25:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:25:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 8 2024-09-13T09:26:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-13T09:26:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:26:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:26:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:26:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:26:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:26:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:26:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-13T09:26:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='test: 2024-09-13T09:26:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:26:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+replaying 2024-09-13T09:26:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219515,"remote_snapshot_timestamp":1726219515,"replay_state":"idle"} 2024-09-13T09:26:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: service: mirror.2 on smithi190 2024-09-13T09:26:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13 09:25:59 2024-09-13T09:26:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2024-09-13T09:26:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2024-09-13T09:26:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+stopped 2024-09-13T09:26:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: local image is primary 2024-09-13T09:26:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13 09:25:54' 2024-09-13T09:26:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ state: .*up+replaying' 2024-09-13T09:26:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: state: up+replaying 2024-09-13T09:26:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ description: .*' 2024-09-13T09:26:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219515,"remote_snapshot_timestamp":1726219515,"replay_state":"idle"} 2024-09-13T09:26:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:26:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2024-09-13T09:26:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^up+' 2024-09-13T09:26:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2024-09-13T09:26:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep 'service: *mirror..* on ' 2024-09-13T09:26:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: service: mirror.2 on smithi190 2024-09-13T09:26:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update 2024-09-13T09:26:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-13T09:26:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update='2024-09-13 09:25:59' 2024-09-13T09:26:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-13 09:25:59' 2024-09-13T09:26:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:26:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:26:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local 'prev_last_update=2024-09-13 09:25:59' 2024-09-13T09:26:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:26:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:26:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-13T09:26:01.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='OKOKOK142094299mirror.0smithi19018.2.4-820-g48a7de7efalseOK42134297mirror.1smithi19018.2.4-820-g48a7de7efalseOK42214293mirror.3smithi19018.2.4-820-g48a7de7efalseOK42674270mirror.2smithi19018.2.4-820-g48a7de7etrueOKtest47e44e3e-e2e8-4660-b7fd-df61813d8d21up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219515,"remote_snapshot_timestamp":1726219515,"replay_state":"idle"}42674270mirror.2smithi1902024-09-13 09:25:59cluster2b8d40c1b-48a7-4e9b-8cbe-df65c34733d8up+stoppedlocal image is primary2024-09-13 09:25:54' 2024-09-13T09:26:01.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update state 2024-09-13T09:26:01.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-13T09:26:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update='2024-09-13 09:25:59' 2024-09-13T09:26:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-13T09:26:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=up+replaying 2024-09-13T09:26:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2024-09-13T09:26:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep up+replaying 2024-09-13T09:26:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2024-09-13T09:26:01.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:26:01.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:26:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap1 2024-09-13T09:26:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap1 2024-09-13T09:26:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:26:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap1 2024-09-13T09:26:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:26:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-13T09:26:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:26:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-13T09:26:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:26:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:09.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:26:09.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:26:09.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:26:09.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:26:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-13T09:26:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:26:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:26:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:26:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:26:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-13T09:26:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:26:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:26:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:26:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:26:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:26:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:26:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:26:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:26:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:26:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:26:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:26:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:26:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:26:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:26:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:26:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:26:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:26:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:26:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:26:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:26:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:26:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:26:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:26:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:26:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:26:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:26:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:26:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:26:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:26:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:26:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:26:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:26:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:26:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:26:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:26:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:26:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:26:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:26:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:26:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:26:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:26:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:26:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:26:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:26:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:26:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:26:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:26:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:26:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:26:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:26:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:26:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:26:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:26:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:26:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:26:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:26:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:26:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:26:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:26:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:26:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:26:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:26:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:26:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:26:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:26:10.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:26:10.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:26:10.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:26:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:26:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:26:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:26:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:26:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:26:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:26:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:26:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:26:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:26:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:26:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:26:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:26:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:26:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:26:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:26:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:26:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:26:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:26:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:26:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:26:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:26:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:26:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:26:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:26:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:26:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:26:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:26:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:26:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:26:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:26:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:26:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:26:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:26:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:26:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:26:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:26:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:26:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:26:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:26:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:26:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:26:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-13T09:26:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:26:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:26:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:26:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:26:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-13T09:26:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:26:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:26:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-test.status 2024-09-13T09:26:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-test.status 2024-09-13T09:26:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:26:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:26:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-test.status 2024-09-13T09:26:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-test.status 2024-09-13T09:26:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:26:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:26:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-13T09:26:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:26:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-13T09:26:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 6 2024-09-13T09:26:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:26:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:26:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:26:11.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-13T09:26:11.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:26:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:26:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=6 2024-09-13T09:26:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:26:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:26:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:26:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:26:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:26:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:26:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:26:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:26:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:26:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:26:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:26:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:26:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:26:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:26:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 6 = '' 2024-09-13T09:26:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:26:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:26:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:26:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:26:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:26:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:26:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:26:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:26:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:26:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:26:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:26:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:26:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:26:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 6 = '' 2024-09-13T09:26:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:26:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:26:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:26:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:26:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:26:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:26:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:26:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:26:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:26:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:26:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:26:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:26:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:26:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 6 = '' 2024-09-13T09:26:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:26:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:26:14.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:26:14.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:26:14.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:14.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:14.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:26:14.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:26:14.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:26:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:26:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=6 2024-09-13T09:26:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 6 = 6 2024-09-13T09:26:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:26:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-09-13T09:26:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:26:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap1 2024-09-13T09:26:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:26:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:26:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:26:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-09-13T09:26:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2024-09-13T09:26:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T09:26:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:26:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:26:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10ae4855b728 2024-09-13T09:26:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10ae4855b728 2024-09-13T09:26:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:26:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:26:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:26:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:26:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:26:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:26:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:26:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:26:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:26:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:26:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:26:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:26:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:26:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:26:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap1 2024-09-13T09:26:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap1 2024-09-13T09:26:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:26:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:26:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:26:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:26:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:26:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:26:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:26:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:26:17.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:26:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:26:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:26:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name=snap1-clean 2024-09-13T09:26:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-09-13T09:26:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:26:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap1-clean 2024-09-13T09:26:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-09-13T09:26:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:26:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror test 2024-09-13T09:26:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:26:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:26:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=6 2024-09-13T09:26:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:26:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-13T09:26:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 70083584 bytes 2024-09-13T09:26:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 104181760 bytes 2024-09-13T09:26:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 129922048 bytes 2024-09-13T09:26:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 161980416 bytes 2024-09-13T09:26:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 216169472 bytes 2024-09-13T09:26:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 248652800 bytes 2024-09-13T09:26:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 269489152 bytes 2024-09-13T09:26:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 328588288 bytes 2024-09-13T09:26:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 350428160 bytes 2024-09-13T09:26:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 397083648 bytes 2024-09-13T09:26:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 420559872 bytes 2024-09-13T09:26:19.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 434352128 bytes 2024-09-13T09:26:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 485948416 bytes 2024-09-13T09:26:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 494188544 bytes 2024-09-13T09:26:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 572101632 bytes 2024-09-13T09:26:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 633532416 bytes 2024-09-13T09:26:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 655257600 bytes 2024-09-13T09:26:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 688656384 bytes 2024-09-13T09:26:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 701952000 bytes 2024-09-13T09:26:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 702560256 bytes 2024-09-13T09:26:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 720974848 bytes 2024-09-13T09:26:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 731097088 bytes 2024-09-13T09:26:20.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 785423360 bytes 2024-09-13T09:26:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 834462720 bytes 2024-09-13T09:26:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 891226112 bytes 2024-09-13T09:26:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 947843072 bytes 2024-09-13T09:26:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1020681216 bytes 2024-09-13T09:26:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1069430784 bytes 2024-09-13T09:26:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1126897664 bytes 2024-09-13T09:26:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1188512768 bytes 2024-09-13T09:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1243675648 bytes 2024-09-13T09:26:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1300597760 bytes 2024-09-13T09:26:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1342364672 bytes 2024-09-13T09:26:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1378383872 bytes 2024-09-13T09:26:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1390931968 bytes 2024-09-13T09:26:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1400869888 bytes 2024-09-13T09:26:22.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1452552192 bytes 2024-09-13T09:26:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1490118656 bytes 2024-09-13T09:26:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1541524480 bytes 2024-09-13T09:26:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1647695872 bytes 2024-09-13T09:26:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1670430720 bytes 2024-09-13T09:26:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1727791104 bytes 2024-09-13T09:26:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1728428032 bytes 2024-09-13T09:26:22.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1739432960 bytes 2024-09-13T09:26:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1765192704 bytes 2024-09-13T09:26:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1806413824 bytes 2024-09-13T09:26:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1807035392 bytes 2024-09-13T09:26:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1865616384 bytes 2024-09-13T09:26:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1905144832 bytes 2024-09-13T09:26:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1921507328 bytes 2024-09-13T09:26:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 1954043904 bytes 2024-09-13T09:26:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 1981868032 bytes 2024-09-13T09:26:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2006039552 bytes 2024-09-13T09:26:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2048097280 bytes 2024-09-13T09:26:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2061496320 bytes 2024-09-13T09:26:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2101825536 bytes 2024-09-13T09:26:23.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2126186496 bytes 2024-09-13T09:26:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2168573952 bytes 2024-09-13T09:26:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2226797568 bytes 2024-09-13T09:26:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2286661632 bytes 2024-09-13T09:26:24.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2385272832 bytes 2024-09-13T09:26:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:26:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:26:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:26:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:26:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-13T09:26:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:26:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:26:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:26:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:26:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:26:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:26:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-13T09:26:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:26:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:26:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:26:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:26:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:26:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:26:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-13T09:26:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='test: 2024-09-13T09:26:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:26:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+replaying 2024-09-13T09:26:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219515,"remote_snapshot_timestamp":1726219515,"replay_state":"idle"} 2024-09-13T09:26:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: service: mirror.2 on smithi190 2024-09-13T09:26:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13 09:25:59 2024-09-13T09:26:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2024-09-13T09:26:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2024-09-13T09:26:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+stopped 2024-09-13T09:26:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: local image is primary 2024-09-13T09:26:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13 09:26:24' 2024-09-13T09:26:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ state: .*up+replaying' 2024-09-13T09:26:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: state: up+replaying 2024-09-13T09:26:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ description: .*' 2024-09-13T09:26:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219515,"remote_snapshot_timestamp":1726219515,"replay_state":"idle"} 2024-09-13T09:26:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:26:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2024-09-13T09:26:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^up+' 2024-09-13T09:26:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2024-09-13T09:26:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep 'service: *mirror..* on ' 2024-09-13T09:26:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: service: mirror.2 on smithi190 2024-09-13T09:26:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update 2024-09-13T09:26:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-13T09:26:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update='2024-09-13 09:25:59' 2024-09-13T09:26:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-13 09:25:59' 2024-09-13T09:26:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:26:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:26:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local 'prev_last_update=2024-09-13 09:25:59' 2024-09-13T09:26:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:26:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:26:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-13T09:26:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='OKOKOK142094299mirror.0smithi19018.2.4-820-g48a7de7efalseOK42134297mirror.1smithi19018.2.4-820-g48a7de7efalseOK42214293mirror.3smithi19018.2.4-820-g48a7de7efalseOK42674270mirror.2smithi19018.2.4-820-g48a7de7etrueOKtest47e44e3e-e2e8-4660-b7fd-df61813d8d21up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219515,"remote_snapshot_timestamp":1726219515,"replay_state":"idle"}42674270mirror.2smithi1902024-09-13 09:25:59cluster2b8d40c1b-48a7-4e9b-8cbe-df65c34733d8up+stoppedlocal image is primary2024-09-13 09:26:24' 2024-09-13T09:26:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update state 2024-09-13T09:26:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-13T09:26:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update='2024-09-13 09:25:59' 2024-09-13T09:26:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-13T09:26:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=up+replaying 2024-09-13T09:26:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2024-09-13T09:26:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep up+replaying 2024-09-13T09:26:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2024-09-13T09:26:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:26:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:26:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap2 2024-09-13T09:26:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap2 2024-09-13T09:26:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:26:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap2 2024-09-13T09:26:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:26:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-13T09:26:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:26:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-13T09:26:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:26:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:26:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:26:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:26:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:26:56.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-13T09:26:56.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:26:56.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:56.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:56.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:26:56.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:26:56.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:26:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-13T09:26:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:26:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:26:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:26:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:26:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:26:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:26:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:26:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:26:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:26:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:26:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:26:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:26:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:26:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:26:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:26:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:26:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:26:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:26:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:26:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:26:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:26:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:26:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:26:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:26:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:26:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:26:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:26:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:26:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:26:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:26:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:26:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:26:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:26:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:26:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:26:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:26:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:26:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:26:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:26:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:26:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:26:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:26:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:26:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:26:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:26:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:26:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:26:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:26:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:26:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:26:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:26:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:26:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:26:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:26:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:26:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:26:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:26:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:26:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:26:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:26:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:26:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:26:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:26:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:26:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:26:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:26:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:26:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:26:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:26:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:26:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:26:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:26:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:26:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:26:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:26:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:26:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:26:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:26:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:26:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:26:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:26:56.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:26:56.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:26:56.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:26:56.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:26:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:26:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:26:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:26:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:26:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:26:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:26:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:26:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:26:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:26:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:26:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:26:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:26:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:26:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:26:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:26:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:26:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:26:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:26:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:26:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:26:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:26:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:26:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:26:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:26:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:26:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:26:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:26:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:26:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:26:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:26:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:26:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:26:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:26:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:26:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:26:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:26:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:26:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:26:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:26:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:26:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:26:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:26:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:26:56.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:26:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:26:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:26:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:26:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-13T09:26:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:26:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:26:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:26:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:26:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-13T09:26:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:26:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:26:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-test.status 2024-09-13T09:26:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-test.status 2024-09-13T09:26:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:26:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:26:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-test.status 2024-09-13T09:26:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-test.status 2024-09-13T09:26:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:26:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:26:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-13T09:26:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:26:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-13T09:26:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 9 2024-09-13T09:26:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:26:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:26:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:26:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-13T09:26:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:26:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:26:58.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=9 2024-09-13T09:26:58.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:26:58.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:26:58.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:26:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:26:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:26:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:26:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:26:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:26:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:26:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:26:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:26:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:26:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:26:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:26:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 9 = '' 2024-09-13T09:26:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:26:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:26:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:26:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:26:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:26:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:26:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:26:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:26:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:26:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:26:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:26:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:26:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:26:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 9 = '' 2024-09-13T09:26:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:26:59.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:26:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:26:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:26:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:26:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:26:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:26:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:26:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:27:00.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:27:00.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:27:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:27:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:27:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:27:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:27:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 9 = '' 2024-09-13T09:27:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:27:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:27:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:27:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:27:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:27:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:27:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:27:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:27:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=9 2024-09-13T09:27:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 9 = 9 2024-09-13T09:27:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:27:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-09-13T09:27:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:27:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap2 2024-09-13T09:27:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:27:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:27:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:27:02.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-09-13T09:27:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2024-09-13T09:27:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T09:27:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:27:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 4 2024-09-13T09:27:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10ae4855b728 2024-09-13T09:27:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10ae4855b728 2024-09-13T09:27:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:27:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:27:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:27:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:27:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:27:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:27:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:27:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:27:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:27:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:27:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:27:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:27:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:27:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snap1-clean ']' 2024-09-13T09:27:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-13T09:27:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap1-clean 2024-09-13T09:27:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:27:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:27:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:27:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:27:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:27:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:27:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:27:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:27:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:27:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:27:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:27:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap2 2024-09-13T09:27:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap2 2024-09-13T09:27:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:27:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:27:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:27:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:27:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:27:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:27:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:27:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:27:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:27:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:27:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:27:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name=snap2-clean 2024-09-13T09:27:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-09-13T09:27:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:27:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap2-clean 2024-09-13T09:27:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-09-13T09:27:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:27:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror test 2024-09-13T09:27:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:27:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:27:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=13 2024-09-13T09:27:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:27:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-13T09:27:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 92149760 bytes 2024-09-13T09:27:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 125756416 bytes 2024-09-13T09:27:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 141302784 bytes 2024-09-13T09:27:06.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 174875648 bytes 2024-09-13T09:27:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 266157056 bytes 2024-09-13T09:27:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 298454016 bytes 2024-09-13T09:27:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 362918912 bytes 2024-09-13T09:27:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 402448384 bytes 2024-09-13T09:27:06.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 428878848 bytes 2024-09-13T09:27:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 434333696 bytes 2024-09-13T09:27:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 486373376 bytes 2024-09-13T09:27:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 486973440 bytes 2024-09-13T09:27:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 558475264 bytes 2024-09-13T09:27:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 567205888 bytes 2024-09-13T09:27:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 602298368 bytes 2024-09-13T09:27:07.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 617010176 bytes 2024-09-13T09:27:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 680716288 bytes 2024-09-13T09:27:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 707990528 bytes 2024-09-13T09:27:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 794949632 bytes 2024-09-13T09:27:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 838886400 bytes 2024-09-13T09:27:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 879134720 bytes 2024-09-13T09:27:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 896112640 bytes 2024-09-13T09:27:08.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 943841280 bytes 2024-09-13T09:27:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1018085376 bytes 2024-09-13T09:27:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1082672128 bytes 2024-09-13T09:27:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1135080448 bytes 2024-09-13T09:27:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1188089856 bytes 2024-09-13T09:27:08.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1264546816 bytes 2024-09-13T09:27:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1320463360 bytes 2024-09-13T09:27:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1348998144 bytes 2024-09-13T09:27:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1378034688 bytes 2024-09-13T09:27:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1405668352 bytes 2024-09-13T09:27:09.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1440960512 bytes 2024-09-13T09:27:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1450465280 bytes 2024-09-13T09:27:09.355 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9257040858076172) > weight (0.25) 2024-09-13T09:27:09.355 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9665849663764212) > weight (0.25) 2024-09-13T09:27:09.355 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-13T09:27:09.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-13T09:27:09.356 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-13T09:27:09.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:27:09.355+0000 7f6456a0a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 36462) UID: 1000 2024-09-13T09:27:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1460525056 bytes 2024-09-13T09:27:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1531215872 bytes 2024-09-13T09:27:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1559468032 bytes 2024-09-13T09:27:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1594901504 bytes 2024-09-13T09:27:09.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1616669696 bytes 2024-09-13T09:27:09.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1657115648 bytes 2024-09-13T09:27:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1667344384 bytes 2024-09-13T09:27:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1694349312 bytes 2024-09-13T09:27:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1719352320 bytes 2024-09-13T09:27:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1743273984 bytes 2024-09-13T09:27:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1771476992 bytes 2024-09-13T09:27:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1820521472 bytes 2024-09-13T09:27:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1839930368 bytes 2024-09-13T09:27:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1842018304 bytes 2024-09-13T09:27:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1846658048 bytes 2024-09-13T09:27:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1880578048 bytes 2024-09-13T09:27:10.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 1915043840 bytes 2024-09-13T09:27:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 1951162368 bytes 2024-09-13T09:27:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2004541440 bytes 2024-09-13T09:27:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2038989824 bytes 2024-09-13T09:27:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2094693376 bytes 2024-09-13T09:27:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2136065024 bytes 2024-09-13T09:27:11.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2211388416 bytes 2024-09-13T09:27:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2266019840 bytes 2024-09-13T09:27:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2333046784 bytes 2024-09-13T09:27:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2361035776 bytes 2024-09-13T09:27:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2400817152 bytes 2024-09-13T09:27:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2420302848 bytes 2024-09-13T09:27:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2422709248 bytes 2024-09-13T09:27:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2457863168 bytes 2024-09-13T09:27:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2489799680 bytes 2024-09-13T09:27:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2576723968 bytes 2024-09-13T09:27:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2635731968 bytes 2024-09-13T09:27:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2673846272 bytes 2024-09-13T09:27:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2683173888 bytes 2024-09-13T09:27:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2715638784 bytes 2024-09-13T09:27:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2720372736 bytes 2024-09-13T09:27:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2751828992 bytes 2024-09-13T09:27:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2761963520 bytes 2024-09-13T09:27:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 2802529280 bytes 2024-09-13T09:27:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 2811438080 bytes 2024-09-13T09:27:13.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 2893813760 bytes 2024-09-13T09:27:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 2920614912 bytes 2024-09-13T09:27:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 2948054016 bytes 2024-09-13T09:27:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3044725760 bytes 2024-09-13T09:27:13.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3113695232 bytes 2024-09-13T09:27:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3145712640 bytes 2024-09-13T09:27:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3157488640 bytes 2024-09-13T09:27:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3206447104 bytes 2024-09-13T09:27:13.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3207048192 bytes 2024-09-13T09:27:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3229446144 bytes 2024-09-13T09:27:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3247792128 bytes 2024-09-13T09:27:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3288134656 bytes 2024-09-13T09:27:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3383568384 bytes 2024-09-13T09:27:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3391036416 bytes 2024-09-13T09:27:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 3472515072 bytes 2024-09-13T09:27:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 3489309696 bytes 2024-09-13T09:27:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 3535608832 bytes 2024-09-13T09:27:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 3618931712 bytes 2024-09-13T09:27:14.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9400 IOs, 3677320192 bytes 2024-09-13T09:27:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9500 IOs, 3686180864 bytes 2024-09-13T09:27:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9600 IOs, 3742961664 bytes 2024-09-13T09:27:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9700 IOs, 3792003072 bytes 2024-09-13T09:27:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9800 IOs, 3811725312 bytes 2024-09-13T09:27:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9900 IOs, 3868953600 bytes 2024-09-13T09:27:15.357 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-13T09:27:15.357 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:27:15.357 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-13T09:27:15.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-13T09:27:15.358 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-13T09:27:15.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-13T09:27:15.361 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2024-09-13T09:27:15.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10000 IOs, 3909480448 bytes 2024-09-13T09:27:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10100 IOs, 3947557888 bytes 2024-09-13T09:27:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10200 IOs, 4031666176 bytes 2024-09-13T09:27:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10300 IOs, 4046168064 bytes 2024-09-13T09:27:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10400 IOs, 4084787200 bytes 2024-09-13T09:27:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10500 IOs, 4125728768 bytes 2024-09-13T09:27:16.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10600 IOs, 4165810176 bytes 2024-09-13T09:27:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10700 IOs, 4187794432 bytes 2024-09-13T09:27:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10800 IOs, 4288061440 bytes 2024-09-13T09:27:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10900 IOs, 4331879424 bytes 2024-09-13T09:27:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11000 IOs, 4369980416 bytes 2024-09-13T09:27:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11100 IOs, 4470347776 bytes 2024-09-13T09:27:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11200 IOs, 4493281280 bytes 2024-09-13T09:27:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11300 IOs, 4497315840 bytes 2024-09-13T09:27:16.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11400 IOs, 4539114496 bytes 2024-09-13T09:27:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11500 IOs, 4539716608 bytes 2024-09-13T09:27:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11600 IOs, 4557568000 bytes 2024-09-13T09:27:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11700 IOs, 4559144960 bytes 2024-09-13T09:27:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11800 IOs, 4593567744 bytes 2024-09-13T09:27:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11900 IOs, 4632443904 bytes 2024-09-13T09:27:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12000 IOs, 4635777024 bytes 2024-09-13T09:27:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12100 IOs, 4720307200 bytes 2024-09-13T09:27:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12200 IOs, 4770598912 bytes 2024-09-13T09:27:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12300 IOs, 4780914688 bytes 2024-09-13T09:27:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12400 IOs, 4810371072 bytes 2024-09-13T09:27:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12500 IOs, 4854377472 bytes 2024-09-13T09:27:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12600 IOs, 4890807296 bytes 2024-09-13T09:27:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12700 IOs, 4943462400 bytes 2024-09-13T09:27:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12800 IOs, 4963448832 bytes 2024-09-13T09:27:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12900 IOs, 4983862272 bytes 2024-09-13T09:27:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:13000 IOs, 5043791872 bytes 2024-09-13T09:27:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:13100 IOs, 5116623872 bytes 2024-09-13T09:27:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:13200 IOs, 5130628096 bytes 2024-09-13T09:27:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:13300 IOs, 5161055232 bytes 2024-09-13T09:27:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:13400 IOs, 5196066816 bytes 2024-09-13T09:27:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:13500 IOs, 5248935936 bytes 2024-09-13T09:27:18.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:13600 IOs, 5345565696 bytes 2024-09-13T09:27:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:13700 IOs, 5391374336 bytes 2024-09-13T09:27:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:27:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:27:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:27:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:27:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-13T09:27:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:27:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:27:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:27:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:27:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:27:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:27:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-13T09:27:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:27:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:27:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:27:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:27:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:27:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:27:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-13T09:27:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='test: 2024-09-13T09:27:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:27:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+replaying 2024-09-13T09:27:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":266865664.0,"last_snapshot_bytes":533731328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219571,"remote_snapshot_timestamp":1726219571,"replay_state":"idle"} 2024-09-13T09:27:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: service: mirror.2 on smithi190 2024-09-13T09:27:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13 09:26:59 2024-09-13T09:27:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2024-09-13T09:27:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2024-09-13T09:27:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+stopped 2024-09-13T09:27:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: local image is primary 2024-09-13T09:27:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13 09:26:54' 2024-09-13T09:27:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ state: .*up+replaying' 2024-09-13T09:27:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: state: up+replaying 2024-09-13T09:27:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ description: .*' 2024-09-13T09:27:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":266865664.0,"last_snapshot_bytes":533731328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219571,"remote_snapshot_timestamp":1726219571,"replay_state":"idle"} 2024-09-13T09:27:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:27:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2024-09-13T09:27:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^up+' 2024-09-13T09:27:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2024-09-13T09:27:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep 'service: *mirror..* on ' 2024-09-13T09:27:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: service: mirror.2 on smithi190 2024-09-13T09:27:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update 2024-09-13T09:27:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-13T09:27:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update='2024-09-13 09:26:59' 2024-09-13T09:27:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-13 09:26:59' 2024-09-13T09:27:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:27:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:27:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local 'prev_last_update=2024-09-13 09:26:59' 2024-09-13T09:27:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:27:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:27:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-13T09:27:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='OKOKOK142094299mirror.0smithi19018.2.4-820-g48a7de7efalseOK42134297mirror.1smithi19018.2.4-820-g48a7de7efalseOK42214293mirror.3smithi19018.2.4-820-g48a7de7efalseOK42674270mirror.2smithi19018.2.4-820-g48a7de7etrueOKtest47e44e3e-e2e8-4660-b7fd-df61813d8d21up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":266865664.0,"last_snapshot_bytes":533731328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219571,"remote_snapshot_timestamp":1726219571,"replay_state":"idle"}42674270mirror.2smithi1902024-09-13 09:26:59cluster2b8d40c1b-48a7-4e9b-8cbe-df65c34733d8up+stoppedlocal image is primary2024-09-13 09:26:54' 2024-09-13T09:27:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update state 2024-09-13T09:27:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-13T09:27:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update='2024-09-13 09:26:59' 2024-09-13T09:27:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-13T09:27:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=up+replaying 2024-09-13T09:27:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2024-09-13T09:27:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep up+replaying 2024-09-13T09:27:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2024-09-13T09:27:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:27:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:27:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap3 2024-09-13T09:27:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap3 2024-09-13T09:27:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:27:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap3 2024-09-13T09:27:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:27:22.356 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5380784112046406) > weight (0.25) 2024-09-13T09:27:22.356 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-13T09:27:22.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-13T09:27:22.357 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-13T09:27:22.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T09:27:22.357+0000 7f03dea34640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 36270) UID: 1000 2024-09-13T09:27:28.357 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-13T09:27:28.358 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:27:28.358 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-13T09:27:28.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-13T09:27:28.358 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-13T09:27:28.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-13T09:27:28.362 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2024-09-13T09:27:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-13T09:27:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:27:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-13T09:27:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:27:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:27:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:27:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:27:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:27:51.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-13T09:27:51.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:27:51.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:51.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:27:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:27:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:27:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-13T09:27:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:27:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:27:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:27:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:27:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:27:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:27:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:27:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:27:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:27:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:27:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:27:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:27:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:27:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:27:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:27:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:27:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:27:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:27:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:27:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:27:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:27:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:27:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:27:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:27:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:27:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:27:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:27:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:27:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:27:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:27:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:27:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:27:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:27:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:27:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:27:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:27:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:27:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:27:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:27:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:27:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:27:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:27:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:27:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:27:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:27:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:27:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:27:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:27:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:27:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:27:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:27:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:27:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:27:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:27:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:27:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:27:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:27:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:27:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:27:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:27:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:27:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:27:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:27:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:27:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:27:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:27:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:27:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:27:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:27:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:27:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:27:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:27:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:27:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:27:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:27:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:27:51.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:27:51.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:27:51.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:27:51.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:27:51.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:27:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:27:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:27:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:27:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:27:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:27:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:27:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:27:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:27:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:27:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:27:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:27:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:27:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:27:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:27:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:27:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:27:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:27:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:27:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:27:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:27:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:27:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:27:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:27:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:27:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:27:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:27:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:27:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:27:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:27:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:27:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:27:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:27:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:27:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:27:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:27:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:27:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:27:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:27:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:27:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:27:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:27:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:27:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:27:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:27:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:27:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:27:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:27:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:27:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:27:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:27:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:27:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-13T09:27:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:27:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:27:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:27:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:27:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-13T09:27:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:27:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:27:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-test.status 2024-09-13T09:27:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-test.status 2024-09-13T09:27:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:27:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:27:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-test.status 2024-09-13T09:27:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-test.status 2024-09-13T09:27:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:27:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:27:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-13T09:27:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:27:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-13T09:27:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 13 2024-09-13T09:27:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:27:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:27:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:27:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-13T09:27:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:27:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:27:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=13 2024-09-13T09:27:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:27:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:27:53.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:27:53.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:27:53.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:27:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:27:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:27:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:27:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:27:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:27:53.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:27:53.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:27:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:27:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:27:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 13 = '' 2024-09-13T09:27:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:27:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:27:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:27:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:27:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:27:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:27:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:27:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:27:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:27:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:27:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:27:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:27:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:27:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 13 = '' 2024-09-13T09:27:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:27:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:27:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:27:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:27:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:27:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:27:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:27:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:27:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:27:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:27:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:27:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:27:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:27:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 13 = '' 2024-09-13T09:27:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:27:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:27:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:27:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:27:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:27:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:27:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:27:56.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:27:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=13 2024-09-13T09:27:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 13 = 13 2024-09-13T09:27:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:27:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-09-13T09:27:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:27:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap3 2024-09-13T09:27:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:27:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:27:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:27:57.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-09-13T09:27:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2024-09-13T09:27:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T09:27:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:27:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 6 2024-09-13T09:27:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10ae4855b728 2024-09-13T09:27:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10ae4855b728 2024-09-13T09:27:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:27:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:27:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:27:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:27:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:27:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:27:03 2024 2024-09-13T09:27:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:27:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:27:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:27:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:27:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:27:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:27:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:27:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snap2-clean ']' 2024-09-13T09:27:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-13T09:27:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap2-clean 2024-09-13T09:27:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:27:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:27:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:27:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:27:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:27:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:27:58.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:27:58.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:27:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:27:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:27:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:27:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap3 2024-09-13T09:27:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:27:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:27:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap3 2024-09-13T09:27:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:27:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:27:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:27:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:27:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:27:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:27:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:28:00.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:28:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:28:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:28:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:28:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name=snap3-clean 2024-09-13T09:28:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-09-13T09:28:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:28:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap3-clean 2024-09-13T09:28:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-09-13T09:28:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:28:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror test 2024-09-13T09:28:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:28:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:28:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=5 2024-09-13T09:28:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:28:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-13T09:28:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 50250752 bytes 2024-09-13T09:28:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 70531072 bytes 2024-09-13T09:28:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 94345216 bytes 2024-09-13T09:28:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 153028608 bytes 2024-09-13T09:28:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 193360896 bytes 2024-09-13T09:28:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 246604800 bytes 2024-09-13T09:28:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 247195648 bytes 2024-09-13T09:28:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 273037312 bytes 2024-09-13T09:28:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 273656832 bytes 2024-09-13T09:28:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 274278400 bytes 2024-09-13T09:28:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 295701504 bytes 2024-09-13T09:28:02.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 311728128 bytes 2024-09-13T09:28:02.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 349327360 bytes 2024-09-13T09:28:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 366818304 bytes 2024-09-13T09:28:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 444416000 bytes 2024-09-13T09:28:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 475282432 bytes 2024-09-13T09:28:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 482349056 bytes 2024-09-13T09:28:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 519719936 bytes 2024-09-13T09:28:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 605300736 bytes 2024-09-13T09:28:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 617250816 bytes 2024-09-13T09:28:03.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 641814528 bytes 2024-09-13T09:28:03.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 643698688 bytes 2024-09-13T09:28:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 687438848 bytes 2024-09-13T09:28:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 731760640 bytes 2024-09-13T09:28:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 768678912 bytes 2024-09-13T09:28:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 795253760 bytes 2024-09-13T09:28:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 843113472 bytes 2024-09-13T09:28:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 926724096 bytes 2024-09-13T09:28:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 984435712 bytes 2024-09-13T09:28:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1087517696 bytes 2024-09-13T09:28:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1102391296 bytes 2024-09-13T09:28:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1117214720 bytes 2024-09-13T09:28:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1159248896 bytes 2024-09-13T09:28:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1295783936 bytes 2024-09-13T09:28:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1323654144 bytes 2024-09-13T09:28:04.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1327080448 bytes 2024-09-13T09:28:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1342114816 bytes 2024-09-13T09:28:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1356332032 bytes 2024-09-13T09:28:05.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1360436224 bytes 2024-09-13T09:28:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1436780544 bytes 2024-09-13T09:28:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1488700416 bytes 2024-09-13T09:28:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1513382912 bytes 2024-09-13T09:28:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1513992192 bytes 2024-09-13T09:28:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1556229120 bytes 2024-09-13T09:28:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1579854848 bytes 2024-09-13T09:28:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1594637312 bytes 2024-09-13T09:28:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1635629056 bytes 2024-09-13T09:28:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1655949312 bytes 2024-09-13T09:28:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1715078144 bytes 2024-09-13T09:28:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1752814592 bytes 2024-09-13T09:28:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 1800458240 bytes 2024-09-13T09:28:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 1856329728 bytes 2024-09-13T09:28:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 1871228928 bytes 2024-09-13T09:28:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:28:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:28:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:28:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:28:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-13T09:28:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:28:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:28:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:28:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:28:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:28:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:28:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-13T09:28:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:28:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:28:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:28:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:28:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:28:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:28:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-13T09:28:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='test: 2024-09-13T09:28:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:28:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+replaying 2024-09-13T09:28:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: replaying, {"bytes_per_second":17755886.93,"bytes_per_snapshot":533981696.0,"last_snapshot_bytes":534557696,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219672,"remote_snapshot_timestamp":1726219672,"replay_state":"idle"} 2024-09-13T09:28:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: service: mirror.2 on smithi190 2024-09-13T09:28:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13 09:27:59 2024-09-13T09:28:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2024-09-13T09:28:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2024-09-13T09:28:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+stopped 2024-09-13T09:28:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: local image is primary 2024-09-13T09:28:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13 09:27:54' 2024-09-13T09:28:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ state: .*up+replaying' 2024-09-13T09:28:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: state: up+replaying 2024-09-13T09:28:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ description: .*' 2024-09-13T09:28:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: description: replaying, {"bytes_per_second":17755886.93,"bytes_per_snapshot":533981696.0,"last_snapshot_bytes":534557696,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219672,"remote_snapshot_timestamp":1726219672,"replay_state":"idle"} 2024-09-13T09:28:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:28:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2024-09-13T09:28:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^up+' 2024-09-13T09:28:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2024-09-13T09:28:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep 'service: *mirror..* on ' 2024-09-13T09:28:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: service: mirror.2 on smithi190 2024-09-13T09:28:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update 2024-09-13T09:28:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-13T09:28:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update='2024-09-13 09:27:59' 2024-09-13T09:28:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-13 09:27:59' 2024-09-13T09:28:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:28:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:28:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local 'prev_last_update=2024-09-13 09:27:59' 2024-09-13T09:28:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:28:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:28:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-13T09:28:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='OKOKOK142094299mirror.0smithi19018.2.4-820-g48a7de7efalseOK42214293mirror.3smithi19018.2.4-820-g48a7de7efalseOK42674270mirror.2smithi19018.2.4-820-g48a7de7etrueOK43854388mirror.1smithi19018.2.4-820-g48a7de7efalseOKtest47e44e3e-e2e8-4660-b7fd-df61813d8d21up+replayingreplaying, {"bytes_per_second":17755886.93,"bytes_per_snapshot":533981696.0,"last_snapshot_bytes":534557696,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219672,"remote_snapshot_timestamp":1726219672,"replay_state":"idle"}42674270mirror.2smithi1902024-09-13 09:27:59cluster2b8d40c1b-48a7-4e9b-8cbe-df65c34733d8up+stoppedlocal image is primary2024-09-13 09:27:54' 2024-09-13T09:28:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update state 2024-09-13T09:28:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-13T09:28:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update='2024-09-13 09:27:59' 2024-09-13T09:28:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-13T09:28:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=up+replaying 2024-09-13T09:28:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2024-09-13T09:28:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep up+replaying 2024-09-13T09:28:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2024-09-13T09:28:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:28:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:28:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap4 2024-09-13T09:28:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap4 2024-09-13T09:28:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:28:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap4 2024-09-13T09:28:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:28:35.363 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-13T09:28:35.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-13T09:28:35.364 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-13T09:28:35.365 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-13T09:28:35.365 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:28:35.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T09:28:35.364+0000 7fcb1a901640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 36253) UID: 1000 2024-09-13T09:28:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-13T09:28:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:28:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-13T09:28:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:28:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:28:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:28:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:28:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:28:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-13T09:28:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:28:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:28:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:28:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:28:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-13T09:28:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:28:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:28:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:28:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:28:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:28:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:28:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:28:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:28:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:28:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:28:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:28:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:28:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:28:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:28:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:28:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:28:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:28:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:28:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:28:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:28:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:28:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:28:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:28:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:28:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:28:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:28:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:28:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:28:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:28:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:28:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:28:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:28:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:28:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:28:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:28:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:28:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:28:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:28:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:28:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:28:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:28:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:28:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:28:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:28:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:28:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:28:38.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-13T09:28:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:28:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:28:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:28:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:28:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:28:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:28:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:28:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:28:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:28:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:28:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:28:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:28:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:28:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:28:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:28:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:28:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:28:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:28:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:28:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:28:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:28:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:28:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:28:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:28:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:28:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:28:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:28:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:28:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:28:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:28:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:28:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:28:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:28:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:28:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:28:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:28:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:28:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:28:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:28:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:28:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:28:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:28:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:28:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:28:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:28:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:28:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:28:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:28:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:28:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:28:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:28:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:28:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:28:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:28:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:28:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:28:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:28:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:28:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:28:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:28:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:28:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:28:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:28:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:28:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:28:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:28:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:28:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:28:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:28:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:28:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:28:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:28:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:28:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:28:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:28:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:28:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:28:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:28:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:28:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:28:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:28:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:28:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:28:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-13T09:28:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:28:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:28:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:28:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:28:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-13T09:28:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:28:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:28:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-test.status 2024-09-13T09:28:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-test.status 2024-09-13T09:28:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:28:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:28:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-test.status 2024-09-13T09:28:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-test.status 2024-09-13T09:28:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:28:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:28:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-13T09:28:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:28:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-13T09:28:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 17 2024-09-13T09:28:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:28:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:28:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:28:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-13T09:28:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:28:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:28:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=17 2024-09-13T09:28:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:28:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:28:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:28:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:28:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:28:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:28:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:28:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:28:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:28:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:28:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:28:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:28:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:28:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:28:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 17 = '' 2024-09-13T09:28:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:28:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:28:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:28:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:28:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:28:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:28:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:28:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:28:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:28:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:28:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:28:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:28:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:28:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 17 = '' 2024-09-13T09:28:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:28:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:28:41.466 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-13T09:28:41.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-13T09:28:41.467 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-13T09:28:41.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-13T09:28:41.470 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2024-09-13T09:28:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:28:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:28:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:28:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:28:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:28:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:28:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:28:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:28:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:28:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:28:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:28:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 17 = '' 2024-09-13T09:28:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:28:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:28:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:28:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:28:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:28:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:28:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:28:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:28:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=17 2024-09-13T09:28:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 17 = 17 2024-09-13T09:28:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:28:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-09-13T09:28:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:28:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap4 2024-09-13T09:28:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:28:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:28:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:28:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-09-13T09:28:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2024-09-13T09:28:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T09:28:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:28:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 8 2024-09-13T09:28:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10ae4855b728 2024-09-13T09:28:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10ae4855b728 2024-09-13T09:28:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:28:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:28:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:28:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:28:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:28:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:27:03 2024 2024-09-13T09:28:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:28:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:28:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:28:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:28:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:28:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:28:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:28:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snap3-clean ']' 2024-09-13T09:28:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-13T09:28:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap3-clean 2024-09-13T09:28:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:28:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:28:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:28:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:28:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:28:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:28:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:28:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:28:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:28:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:28:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:28:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap4 2024-09-13T09:28:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap4 2024-09-13T09:28:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:28:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:28:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:28:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:28:46.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:28:46.362 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.607089060056501) > weight (0.25) 2024-09-13T09:28:46.362 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-13T09:28:46.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-13T09:28:46.363 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-13T09:28:46.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:28:46.362+0000 7f5230ab1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 36421) UID: 1000 2024-09-13T09:28:46.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:28:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:28:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:28:47.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:28:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:28:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:28:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name=snap4-clean 2024-09-13T09:28:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-09-13T09:28:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:28:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap4-clean 2024-09-13T09:28:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-09-13T09:28:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:28:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror test 2024-09-13T09:28:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:28:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:28:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:28:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:28:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=14 2024-09-13T09:28:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:28:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-13T09:28:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 63292416 bytes 2024-09-13T09:28:48.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 107140096 bytes 2024-09-13T09:28:48.364 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-13T09:28:48.364 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:28:48.364 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-13T09:28:48.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-13T09:28:48.365 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-13T09:28:48.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-13T09:28:48.368 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-09-13T09:28:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 138224640 bytes 2024-09-13T09:28:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 177715200 bytes 2024-09-13T09:28:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 221335552 bytes 2024-09-13T09:28:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 221952000 bytes 2024-09-13T09:28:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 235893760 bytes 2024-09-13T09:28:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 248503296 bytes 2024-09-13T09:28:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 322228224 bytes 2024-09-13T09:28:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 353026048 bytes 2024-09-13T09:28:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 435154944 bytes 2024-09-13T09:28:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 513039360 bytes 2024-09-13T09:28:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 513628160 bytes 2024-09-13T09:28:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 601971712 bytes 2024-09-13T09:28:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 616419328 bytes 2024-09-13T09:28:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 635167744 bytes 2024-09-13T09:28:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 664049664 bytes 2024-09-13T09:28:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 700490752 bytes 2024-09-13T09:28:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 713843712 bytes 2024-09-13T09:28:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 801241088 bytes 2024-09-13T09:28:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 825839616 bytes 2024-09-13T09:28:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 878073856 bytes 2024-09-13T09:28:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 903479296 bytes 2024-09-13T09:28:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 940180480 bytes 2024-09-13T09:28:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 957181952 bytes 2024-09-13T09:28:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1001216000 bytes 2024-09-13T09:28:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1043483648 bytes 2024-09-13T09:28:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1085244416 bytes 2024-09-13T09:28:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1132607488 bytes 2024-09-13T09:28:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1147212800 bytes 2024-09-13T09:28:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1172548608 bytes 2024-09-13T09:28:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1206455296 bytes 2024-09-13T09:28:51.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1302160384 bytes 2024-09-13T09:28:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1349242880 bytes 2024-09-13T09:28:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1372289024 bytes 2024-09-13T09:28:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1472123904 bytes 2024-09-13T09:28:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1494193152 bytes 2024-09-13T09:28:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1544444928 bytes 2024-09-13T09:28:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1552633856 bytes 2024-09-13T09:28:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1629364224 bytes 2024-09-13T09:28:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1715110912 bytes 2024-09-13T09:28:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1786916864 bytes 2024-09-13T09:28:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1795800064 bytes 2024-09-13T09:28:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1870568448 bytes 2024-09-13T09:28:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1926079488 bytes 2024-09-13T09:28:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1938011136 bytes 2024-09-13T09:28:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2021521408 bytes 2024-09-13T09:28:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2102121472 bytes 2024-09-13T09:28:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2136980480 bytes 2024-09-13T09:28:53.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2182497280 bytes 2024-09-13T09:28:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2222266368 bytes 2024-09-13T09:28:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2281606144 bytes 2024-09-13T09:28:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2336620544 bytes 2024-09-13T09:28:53.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2397071360 bytes 2024-09-13T09:28:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2475514880 bytes 2024-09-13T09:28:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2536862720 bytes 2024-09-13T09:28:53.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2583831552 bytes 2024-09-13T09:28:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2663388160 bytes 2024-09-13T09:28:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2675709952 bytes 2024-09-13T09:28:54.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2712524800 bytes 2024-09-13T09:28:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2737955840 bytes 2024-09-13T09:28:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2794460160 bytes 2024-09-13T09:28:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2895471616 bytes 2024-09-13T09:28:54.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2920039424 bytes 2024-09-13T09:28:54.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2984701952 bytes 2024-09-13T09:28:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 3006866432 bytes 2024-09-13T09:28:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 3065036800 bytes 2024-09-13T09:28:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 3087418368 bytes 2024-09-13T09:28:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 3158351872 bytes 2024-09-13T09:28:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 3200188416 bytes 2024-09-13T09:28:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 3223123968 bytes 2024-09-13T09:28:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 3257816064 bytes 2024-09-13T09:28:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 3318038528 bytes 2024-09-13T09:28:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3320200192 bytes 2024-09-13T09:28:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3386858496 bytes 2024-09-13T09:28:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3450166272 bytes 2024-09-13T09:28:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3525368832 bytes 2024-09-13T09:28:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3556067328 bytes 2024-09-13T09:28:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3626521600 bytes 2024-09-13T09:28:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3712387072 bytes 2024-09-13T09:28:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3782488064 bytes 2024-09-13T09:28:56.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3865080832 bytes 2024-09-13T09:28:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3867919360 bytes 2024-09-13T09:28:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3931020288 bytes 2024-09-13T09:28:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3942739968 bytes 2024-09-13T09:28:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 4002959360 bytes 2024-09-13T09:28:57.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 4036254720 bytes 2024-09-13T09:28:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 4063244288 bytes 2024-09-13T09:28:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 4094199808 bytes 2024-09-13T09:28:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 4147780608 bytes 2024-09-13T09:28:57.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 4199148544 bytes 2024-09-13T09:28:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 4251645952 bytes 2024-09-13T09:28:57.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 4313620480 bytes 2024-09-13T09:28:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9400 IOs, 4343250944 bytes 2024-09-13T09:28:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9500 IOs, 4388192256 bytes 2024-09-13T09:28:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9600 IOs, 4413364224 bytes 2024-09-13T09:28:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9700 IOs, 4439815168 bytes 2024-09-13T09:28:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9800 IOs, 4508165120 bytes 2024-09-13T09:28:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9900 IOs, 4508784640 bytes 2024-09-13T09:28:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10000 IOs, 4549222400 bytes 2024-09-13T09:28:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10100 IOs, 4586762240 bytes 2024-09-13T09:28:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10200 IOs, 4632235008 bytes 2024-09-13T09:28:58.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10300 IOs, 4685050880 bytes 2024-09-13T09:28:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10400 IOs, 4736398336 bytes 2024-09-13T09:28:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10500 IOs, 4766329856 bytes 2024-09-13T09:28:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10600 IOs, 4823988224 bytes 2024-09-13T09:28:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10700 IOs, 4836903936 bytes 2024-09-13T09:28:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10800 IOs, 4936594432 bytes 2024-09-13T09:28:59.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10900 IOs, 4964164608 bytes 2024-09-13T09:28:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11000 IOs, 5014124544 bytes 2024-09-13T09:28:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11100 IOs, 5014727680 bytes 2024-09-13T09:28:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11200 IOs, 5120788480 bytes 2024-09-13T09:28:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11300 IOs, 5175009280 bytes 2024-09-13T09:28:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11400 IOs, 5196923904 bytes 2024-09-13T09:28:59.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11500 IOs, 5208953856 bytes 2024-09-13T09:29:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11600 IOs, 5275586560 bytes 2024-09-13T09:29:00.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11700 IOs, 5303452672 bytes 2024-09-13T09:29:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11800 IOs, 5349468160 bytes 2024-09-13T09:29:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11900 IOs, 5362322432 bytes 2024-09-13T09:29:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12000 IOs, 5397365760 bytes 2024-09-13T09:29:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12100 IOs, 5423121408 bytes 2024-09-13T09:29:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12200 IOs, 5464744960 bytes 2024-09-13T09:29:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12300 IOs, 5480171520 bytes 2024-09-13T09:29:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12400 IOs, 5517870080 bytes 2024-09-13T09:29:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12500 IOs, 5547862016 bytes 2024-09-13T09:29:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12600 IOs, 5554204672 bytes 2024-09-13T09:29:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12700 IOs, 5574941696 bytes 2024-09-13T09:29:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12800 IOs, 5671993344 bytes 2024-09-13T09:29:01.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12900 IOs, 5744936960 bytes 2024-09-13T09:29:01.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:13000 IOs, 5839469568 bytes 2024-09-13T09:29:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:13100 IOs, 5900224512 bytes 2024-09-13T09:29:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:13200 IOs, 6023359488 bytes 2024-09-13T09:29:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:13300 IOs, 6074298368 bytes 2024-09-13T09:29:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:13400 IOs, 6140140544 bytes 2024-09-13T09:29:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:29:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:29:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:29:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:29:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-13T09:29:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:29:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:29:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:29:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:29:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:29:02.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:29:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-13T09:29:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:29:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:29:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:29:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:29:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:29:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:29:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-13T09:29:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='test: 2024-09-13T09:29:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:29:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+replaying 2024-09-13T09:29:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: replaying, {"bytes_per_second":17744384.0,"bytes_per_snapshot":533444608.0,"last_snapshot_bytes":532331520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219719,"remote_snapshot_timestamp":1726219719,"replay_state":"idle"} 2024-09-13T09:29:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: service: mirror.2 on smithi190 2024-09-13T09:29:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13 09:28:59 2024-09-13T09:29:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2024-09-13T09:29:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2024-09-13T09:29:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+stopped 2024-09-13T09:29:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: local image is primary 2024-09-13T09:29:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13 09:28:54' 2024-09-13T09:29:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ state: .*up+replaying' 2024-09-13T09:29:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: state: up+replaying 2024-09-13T09:29:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ description: .*' 2024-09-13T09:29:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: description: replaying, {"bytes_per_second":17744384.0,"bytes_per_snapshot":533444608.0,"last_snapshot_bytes":532331520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219719,"remote_snapshot_timestamp":1726219719,"replay_state":"idle"} 2024-09-13T09:29:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:29:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2024-09-13T09:29:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^up+' 2024-09-13T09:29:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2024-09-13T09:29:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep 'service: *mirror..* on ' 2024-09-13T09:29:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: service: mirror.2 on smithi190 2024-09-13T09:29:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update 2024-09-13T09:29:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-13T09:29:03.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update='2024-09-13 09:28:59' 2024-09-13T09:29:03.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-13 09:28:59' 2024-09-13T09:29:03.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:29:03.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:03.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:03.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:29:03.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local 'prev_last_update=2024-09-13 09:28:59' 2024-09-13T09:29:03.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:29:03.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:29:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-13T09:29:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='OKOKOK142214293mirror.3smithi19018.2.4-820-g48a7de7efalseOK42674270mirror.2smithi19018.2.4-820-g48a7de7etrueOK43854388mirror.1smithi19018.2.4-820-g48a7de7efalseOK44274430mirror.0smithi19018.2.4-820-g48a7de7efalseOKtest47e44e3e-e2e8-4660-b7fd-df61813d8d21up+replayingreplaying, {"bytes_per_second":17744384.0,"bytes_per_snapshot":533444608.0,"last_snapshot_bytes":532331520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219719,"remote_snapshot_timestamp":1726219719,"replay_state":"idle"}42674270mirror.2smithi1902024-09-13 09:28:59cluster2b8d40c1b-48a7-4e9b-8cbe-df65c34733d8up+stoppedlocal image is primary2024-09-13 09:28:54' 2024-09-13T09:29:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update state 2024-09-13T09:29:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-13T09:29:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update='2024-09-13 09:28:59' 2024-09-13T09:29:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-13T09:29:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=up+replaying 2024-09-13T09:29:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2024-09-13T09:29:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep up+replaying 2024-09-13T09:29:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2024-09-13T09:29:03.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:29:03.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:29:03.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap5 2024-09-13T09:29:03.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap5 2024-09-13T09:29:03.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:29:03.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:03.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:03.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap5 2024-09-13T09:29:03.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:29:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-13T09:29:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:29:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-13T09:29:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:29:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:29:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:29:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:29:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:29:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-13T09:29:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:29:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:29:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:29:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:29:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-13T09:29:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:29:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:29:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:29:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:29:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:29:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:29:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:29:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:29:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:29:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:29:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:29:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:29:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:29:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:29:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:29:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:29:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:29:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:29:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:29:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:29:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:29:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:29:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:29:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:29:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:29:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:29:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:29:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:29:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:29:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:29:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:29:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:29:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:29:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:29:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:29:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:29:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:29:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:29:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:29:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:29:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:29:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:29:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:29:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:29:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:29:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:29:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:29:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:29:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:29:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:29:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:29:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:29:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:29:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:29:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:29:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:29:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:29:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:29:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:29:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:29:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:29:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:29:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:29:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:29:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:29:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:29:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:29:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:29:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:29:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:29:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:29:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:29:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:29:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:29:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:29:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:29:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:29:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:29:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:29:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:29:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:29:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:29:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:29:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:29:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:29:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:29:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:29:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:29:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:29:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:29:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:29:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:29:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:29:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:29:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:29:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:29:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:29:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:29:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:29:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:29:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:29:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:29:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:29:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:29:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:29:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:29:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:29:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:29:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:29:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:29:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:29:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:29:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:29:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:29:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:29:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:29:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:29:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:29:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:29:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:29:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:29:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:29:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:29:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:29:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:29:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:29:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:29:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:29:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:29:33.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:29:33.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:29:33.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:29:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-13T09:29:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:29:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:29:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:29:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:29:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-13T09:29:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:29:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:29:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-test.status 2024-09-13T09:29:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-test.status 2024-09-13T09:29:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:29:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:29:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-test.status 2024-09-13T09:29:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-test.status 2024-09-13T09:29:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:29:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:29:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-13T09:29:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:29:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-13T09:29:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 21 2024-09-13T09:29:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:29:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:29:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:29:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-13T09:29:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:29:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:29:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=21 2024-09-13T09:29:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:29:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:29:35.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:29:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:29:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:29:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:29:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:29:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:29:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:29:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:29:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:29:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:29:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:29:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:29:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 21 = '' 2024-09-13T09:29:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:29:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:29:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:29:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:29:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:29:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:29:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:29:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:29:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:29:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:29:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:29:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:29:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:29:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 21 = '' 2024-09-13T09:29:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:29:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:29:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:29:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:29:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:29:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:29:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:29:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:29:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:29:37.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:29:37.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:29:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:29:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:29:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 21 = '' 2024-09-13T09:29:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:29:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:29:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:29:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:29:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:29:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:29:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:29:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:29:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=21 2024-09-13T09:29:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 21 = 21 2024-09-13T09:29:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:29:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-09-13T09:29:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:29:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap5 2024-09-13T09:29:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:29:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:29:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:29:39.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-09-13T09:29:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2024-09-13T09:29:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T09:29:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:29:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 10 2024-09-13T09:29:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10ae4855b728 2024-09-13T09:29:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10ae4855b728 2024-09-13T09:29:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:29:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:29:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:29:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:29:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:29:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:28:45 2024 2024-09-13T09:29:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:29:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:29:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:29:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:29:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:29:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:29:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:29:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snap4-clean ']' 2024-09-13T09:29:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-13T09:29:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap4-clean 2024-09-13T09:29:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:29:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:29:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:29:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:29:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:29:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:29:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:29:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:29:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:29:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:29:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:29:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap5 2024-09-13T09:29:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap5 2024-09-13T09:29:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:29:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:29:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:29:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:29:41.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:29:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:29:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:29:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:29:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:29:42.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:29:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:29:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name=snap5-clean 2024-09-13T09:29:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-09-13T09:29:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:29:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap5-clean 2024-09-13T09:29:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-09-13T09:29:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:29:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror test 2024-09-13T09:29:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:29:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:29:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=14 2024-09-13T09:29:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:29:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-13T09:29:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 39022592 bytes 2024-09-13T09:29:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 111257600 bytes 2024-09-13T09:29:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 148482048 bytes 2024-09-13T09:29:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 167321600 bytes 2024-09-13T09:29:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 180798464 bytes 2024-09-13T09:29:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 212858880 bytes 2024-09-13T09:29:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 292152320 bytes 2024-09-13T09:29:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 292763648 bytes 2024-09-13T09:29:44.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 293390336 bytes 2024-09-13T09:29:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 371703808 bytes 2024-09-13T09:29:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 443195392 bytes 2024-09-13T09:29:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 471200768 bytes 2024-09-13T09:29:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 543894528 bytes 2024-09-13T09:29:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 605955072 bytes 2024-09-13T09:29:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 630939648 bytes 2024-09-13T09:29:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 725161984 bytes 2024-09-13T09:29:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 819229696 bytes 2024-09-13T09:29:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 878599168 bytes 2024-09-13T09:29:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 944727040 bytes 2024-09-13T09:29:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 967458816 bytes 2024-09-13T09:29:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1027861504 bytes 2024-09-13T09:29:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1048433664 bytes 2024-09-13T09:29:46.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1081534464 bytes 2024-09-13T09:29:46.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1149382656 bytes 2024-09-13T09:29:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1209109504 bytes 2024-09-13T09:29:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1287222272 bytes 2024-09-13T09:29:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1321593856 bytes 2024-09-13T09:29:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1377599488 bytes 2024-09-13T09:29:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1405733888 bytes 2024-09-13T09:29:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1452185600 bytes 2024-09-13T09:29:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1474473984 bytes 2024-09-13T09:29:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1555472384 bytes 2024-09-13T09:29:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1619442688 bytes 2024-09-13T09:29:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1643103232 bytes 2024-09-13T09:29:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1709399040 bytes 2024-09-13T09:29:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1711545344 bytes 2024-09-13T09:29:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1733758976 bytes 2024-09-13T09:29:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1760852992 bytes 2024-09-13T09:29:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1767936000 bytes 2024-09-13T09:29:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1768566784 bytes 2024-09-13T09:29:47.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1816590336 bytes 2024-09-13T09:29:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1853435904 bytes 2024-09-13T09:29:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1897257984 bytes 2024-09-13T09:29:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1901038592 bytes 2024-09-13T09:29:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1936649216 bytes 2024-09-13T09:29:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1958993920 bytes 2024-09-13T09:29:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2017251328 bytes 2024-09-13T09:29:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2026772480 bytes 2024-09-13T09:29:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2069862400 bytes 2024-09-13T09:29:48.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2153749504 bytes 2024-09-13T09:29:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2215684096 bytes 2024-09-13T09:29:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2254213120 bytes 2024-09-13T09:29:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2355384320 bytes 2024-09-13T09:29:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2396747776 bytes 2024-09-13T09:29:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2506370048 bytes 2024-09-13T09:29:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2519192576 bytes 2024-09-13T09:29:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2540110848 bytes 2024-09-13T09:29:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2591645696 bytes 2024-09-13T09:29:49.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2635809792 bytes 2024-09-13T09:29:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2698285056 bytes 2024-09-13T09:29:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2783490048 bytes 2024-09-13T09:29:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2895662080 bytes 2024-09-13T09:29:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2934754304 bytes 2024-09-13T09:29:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2936899584 bytes 2024-09-13T09:29:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2999279616 bytes 2024-09-13T09:29:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 3060013056 bytes 2024-09-13T09:29:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 3091579904 bytes 2024-09-13T09:29:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 3159934976 bytes 2024-09-13T09:29:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 3214889984 bytes 2024-09-13T09:29:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 3252033536 bytes 2024-09-13T09:29:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 3286851584 bytes 2024-09-13T09:29:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 3314809856 bytes 2024-09-13T09:29:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 3326476288 bytes 2024-09-13T09:29:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3366124544 bytes 2024-09-13T09:29:51.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3451716608 bytes 2024-09-13T09:29:51.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3500819456 bytes 2024-09-13T09:29:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3566923776 bytes 2024-09-13T09:29:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3567551488 bytes 2024-09-13T09:29:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3624956928 bytes 2024-09-13T09:29:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3656438784 bytes 2024-09-13T09:29:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3755001856 bytes 2024-09-13T09:29:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3794849792 bytes 2024-09-13T09:29:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3850507264 bytes 2024-09-13T09:29:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3876982784 bytes 2024-09-13T09:29:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3902879744 bytes 2024-09-13T09:29:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3928139776 bytes 2024-09-13T09:29:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3965625344 bytes 2024-09-13T09:29:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3992134656 bytes 2024-09-13T09:29:52.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 4017454080 bytes 2024-09-13T09:29:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 4062139392 bytes 2024-09-13T09:29:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 4123621376 bytes 2024-09-13T09:29:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 4205975552 bytes 2024-09-13T09:29:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 4257129472 bytes 2024-09-13T09:29:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9400 IOs, 4300282880 bytes 2024-09-13T09:29:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9500 IOs, 4382769152 bytes 2024-09-13T09:29:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9600 IOs, 4425778176 bytes 2024-09-13T09:29:53.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9700 IOs, 4450855936 bytes 2024-09-13T09:29:53.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9800 IOs, 4459892736 bytes 2024-09-13T09:29:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9900 IOs, 4490143744 bytes 2024-09-13T09:29:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10000 IOs, 4502682624 bytes 2024-09-13T09:29:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10100 IOs, 4538668032 bytes 2024-09-13T09:29:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10200 IOs, 4604340224 bytes 2024-09-13T09:29:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10300 IOs, 4648867840 bytes 2024-09-13T09:29:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10400 IOs, 4683194368 bytes 2024-09-13T09:29:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10500 IOs, 4695312384 bytes 2024-09-13T09:29:54.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10600 IOs, 4716187648 bytes 2024-09-13T09:29:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10700 IOs, 4740930560 bytes 2024-09-13T09:29:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10800 IOs, 4809348096 bytes 2024-09-13T09:29:54.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10900 IOs, 4832101376 bytes 2024-09-13T09:29:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11000 IOs, 4911126528 bytes 2024-09-13T09:29:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11100 IOs, 4956701696 bytes 2024-09-13T09:29:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11200 IOs, 4972940288 bytes 2024-09-13T09:29:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11300 IOs, 5073422336 bytes 2024-09-13T09:29:55.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11400 IOs, 5147685888 bytes 2024-09-13T09:29:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11500 IOs, 5219252224 bytes 2024-09-13T09:29:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11600 IOs, 5226842112 bytes 2024-09-13T09:29:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11700 IOs, 5249092608 bytes 2024-09-13T09:29:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11800 IOs, 5287147520 bytes 2024-09-13T09:29:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11900 IOs, 5311787008 bytes 2024-09-13T09:29:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12000 IOs, 5321720832 bytes 2024-09-13T09:29:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12100 IOs, 5364976640 bytes 2024-09-13T09:29:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12200 IOs, 5451245568 bytes 2024-09-13T09:29:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12300 IOs, 5539096576 bytes 2024-09-13T09:29:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12400 IOs, 5604218880 bytes 2024-09-13T09:29:56.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12500 IOs, 5617728512 bytes 2024-09-13T09:29:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12600 IOs, 5650173952 bytes 2024-09-13T09:29:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12700 IOs, 5717761024 bytes 2024-09-13T09:29:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12800 IOs, 5785307136 bytes 2024-09-13T09:29:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12900 IOs, 5820839936 bytes 2024-09-13T09:29:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:13000 IOs, 5896555520 bytes 2024-09-13T09:29:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:29:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:29:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:29:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:29:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-13T09:29:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:29:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:29:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:29:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:29:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:29:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:29:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-13T09:29:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:29:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:29:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:29:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:29:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:29:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:29:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-13T09:29:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='test: 2024-09-13T09:29:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:29:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+replaying 2024-09-13T09:29:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533444608.0,"last_snapshot_bytes":532331520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219719,"remote_snapshot_timestamp":1726219719,"replay_state":"idle"} 2024-09-13T09:29:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: service: mirror.2 on smithi190 2024-09-13T09:29:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13 09:29:29 2024-09-13T09:29:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2024-09-13T09:29:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2024-09-13T09:29:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+stopped 2024-09-13T09:29:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: local image is primary 2024-09-13T09:29:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13 09:29:54' 2024-09-13T09:29:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ state: .*up+replaying' 2024-09-13T09:29:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: state: up+replaying 2024-09-13T09:29:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ description: .*' 2024-09-13T09:29:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533444608.0,"last_snapshot_bytes":532331520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219719,"remote_snapshot_timestamp":1726219719,"replay_state":"idle"} 2024-09-13T09:29:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:29:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2024-09-13T09:29:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^up+' 2024-09-13T09:29:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2024-09-13T09:29:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep 'service: *mirror..* on ' 2024-09-13T09:29:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: service: mirror.2 on smithi190 2024-09-13T09:29:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update 2024-09-13T09:29:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-13T09:29:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update='2024-09-13 09:29:29' 2024-09-13T09:29:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-13 09:29:29' 2024-09-13T09:29:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:29:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:29:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local 'prev_last_update=2024-09-13 09:29:29' 2024-09-13T09:29:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:29:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:29:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-13T09:29:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='OKOKOK142214293mirror.3smithi19018.2.4-820-g48a7de7efalseOK42674270mirror.2smithi19018.2.4-820-g48a7de7etrueOK43854388mirror.1smithi19018.2.4-820-g48a7de7efalseOK44274430mirror.0smithi19018.2.4-820-g48a7de7efalseOKtest47e44e3e-e2e8-4660-b7fd-df61813d8d21up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533444608.0,"last_snapshot_bytes":532331520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219719,"remote_snapshot_timestamp":1726219719,"replay_state":"idle"}42674270mirror.2smithi1902024-09-13 09:29:29cluster2b8d40c1b-48a7-4e9b-8cbe-df65c34733d8up+stoppedlocal image is primary2024-09-13 09:29:54' 2024-09-13T09:29:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update state 2024-09-13T09:29:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-13T09:29:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update='2024-09-13 09:29:29' 2024-09-13T09:29:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-13T09:29:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=up+replaying 2024-09-13T09:29:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2024-09-13T09:29:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep up+replaying 2024-09-13T09:29:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2024-09-13T09:29:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:29:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:29:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap6 2024-09-13T09:29:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap6 2024-09-13T09:29:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:29:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:29:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:29:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap6 2024-09-13T09:29:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:30:24.368 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-13T09:30:24.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-13T09:30:24.370 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-13T09:30:24.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:30:24.369+0000 7f353d3b8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 37333) UID: 1000 2024-09-13T09:30:25.471 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.2879769507630494) > weight (0.25) 2024-09-13T09:30:25.471 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9814967786626134) > weight (0.25) 2024-09-13T09:30:25.471 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-13T09:30:25.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-13T09:30:25.471 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-13T09:30:25.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T09:30:25.471+0000 7f7f5e834640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 37457) UID: 1000 2024-09-13T09:30:27.370 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-13T09:30:27.370 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:30:27.371 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-13T09:30:27.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-13T09:30:27.371 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-13T09:30:27.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-13T09:30:27.375 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2024-09-13T09:30:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-13T09:30:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:30:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:30:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:30:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-13T09:30:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:30:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:30:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:30:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:30:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:30:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:30:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:30:28.472 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-13T09:30:28.472 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:30:28.473 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-13T09:30:28.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-13T09:30:28.473 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-13T09:30:28.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-13T09:30:28.476 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2024-09-13T09:30:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-13T09:30:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:30:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:30:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:30:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:30:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:30:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:30:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-13T09:30:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:30:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:30:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:30:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:30:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:30:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:30:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:30:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:30:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:30:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:30:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:30:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:30:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:30:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:30:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:30:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:30:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:30:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:30:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:30:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:30:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:30:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:30:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:30:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:30:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:30:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:30:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:30:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:30:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:30:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:30:29.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:29.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:29.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:30:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:30:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:30:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:30:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:30:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:30:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:30:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:30:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:30:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:30:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:30:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:30:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:30:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:30:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:30:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:30:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:30:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:30:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:30:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:30:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:30:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:30:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:30:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:30:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:30:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:30:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:30:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:30:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:30:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:30:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:30:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:30:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:30:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:30:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:30:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:30:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:30:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:30:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:30:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:30:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:30:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:30:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-13T09:30:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:30:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:30:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:30:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:30:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:30:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:30:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:30:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:30:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:30:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:30:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:30:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:30:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:30:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:30:29.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:30:29.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:30:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:30:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:30:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:30:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-13T09:30:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:30:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 1 2024-09-13T09:30:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:30:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:30:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:30:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:30:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:30:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:30:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:30:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:30:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:30:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:30:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:30:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:30:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:30:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:30:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:30:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:30:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:30:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:30:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:30:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:30:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:30:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:30:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:30:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:30:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:30:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:30:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:30:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:30:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:30:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:30:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:30:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:30:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:30:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:30:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:30:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:30:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:30:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:30:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:30:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:30:30.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:30:30.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:30:30.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:30:30.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:30:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:30:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:30:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:30:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:30:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:30:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:30:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:30:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:30:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:30:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:30:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:30:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:30:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:30:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:30:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:30:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:30:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:30:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:30:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:30:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:30:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:30:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:31.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:31.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-13T09:30:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:30:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:30:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:30:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:30:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:30:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:30:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:30:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:30:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:30:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:30:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:30:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:30:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:30:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:30:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:30:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:30:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:30:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:30:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:30:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-13T09:30:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:30:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 2 2024-09-13T09:30:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:30:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:30:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:30:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:30:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:30:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:30:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:30:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:30:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:30:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:30:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:30:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:30:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:30:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:30:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:30:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:30:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:30:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:30:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:30:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:30:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:30:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:30:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:30:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:30:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:30:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:30:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:30:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:30:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:30:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:30:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:30:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:30:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:30:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:30:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:30:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:30:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:30:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:30:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:30:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:30:33.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:30:33.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:30:33.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:30:33.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:30:33.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:33.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:33.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:33.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:33.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:30:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:30:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:30:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:30:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:30:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:30:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:30:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:30:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:30:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:30:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:30:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:30:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:30:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:30:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:30:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:30:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:30:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:30:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:30:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:30:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:30:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-13T09:30:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:30:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:30:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:30:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:30:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:30:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:30:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:30:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:30:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:30:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:30:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:30:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:30:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:30:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:30:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:30:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:30:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:30:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:30:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:30:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-13T09:30:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:33.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:30:33.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 4 2024-09-13T09:30:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:30:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:30:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:30:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:30:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:30:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:30:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:30:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:30:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:30:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:30:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:30:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:30:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:30:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:30:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:30:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:30:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:30:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:30:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:30:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:30:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:30:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:30:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:30:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:30:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:30:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:30:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:30:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:30:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:30:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:30:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:30:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:30:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:30:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:30:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:30:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:30:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:30:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:30:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:30:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:30:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:30:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:30:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:30:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:30:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:30:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:30:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:30:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:30:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:30:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:30:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:30:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:30:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:30:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:30:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:30:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:30:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:30:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:30:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:30:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:30:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:30:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:30:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:30:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:30:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:30:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-13T09:30:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:30:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:30:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:30:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:30:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:30:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:30:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:30:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:30:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:30:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:30:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:30:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:30:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:30:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:30:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:30:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:30:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:30:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:30:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:30:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-13T09:30:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:30:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 8 2024-09-13T09:30:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:30:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:30:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:30:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:30:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:30:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:30:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:30:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:30:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:30:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:30:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:30:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:30:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:30:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:30:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:30:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:30:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:30:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:30:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:30:46.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:30:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:30:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:30:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:30:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:30:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:30:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:30:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:30:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:30:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:30:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:30:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:30:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:30:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:30:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:30:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:30:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:30:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:30:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:30:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:30:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:30:46.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:30:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:30:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:30:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:30:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:30:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:30:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:30:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:30:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:30:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:30:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:30:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:30:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:30:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:30:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:30:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:30:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:30:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:30:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:30:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:30:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:30:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:30:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:30:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:30:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:30:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-13T09:30:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:30:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:30:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:30:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:30:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:30:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:30:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:30:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:30:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:30:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:30:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:30:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:30:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:30:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:30:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:30:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:30:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:30:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:30:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:30:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-13T09:30:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:30:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 8 2024-09-13T09:30:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:30:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:30:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:30:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:30:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:30:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:30:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:30:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:30:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:30:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:30:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:30:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:30:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:30:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:30:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:30:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:30:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:30:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:30:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:30:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:30:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:30:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:30:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:30:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:30:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:30:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:30:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:30:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:30:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:30:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:30:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:30:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:30:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:30:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:30:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:30:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:30:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:30:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:30:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:30:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:30:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:30:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:30:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:30:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:30:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:30:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:30:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:30:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:30:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:30:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:30:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:30:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:30:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:30:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:30:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:30:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:30:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:30:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:30:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:30:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:30:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:30:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:30:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:30:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:30:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:30:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:30:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-13T09:30:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:30:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:30:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:30:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:30:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:30:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:30:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:30:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:30:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:30:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:30:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:30:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:30:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:30:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:30:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:30:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:30:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:30:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:30:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:30:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:30:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:30:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:30:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:30:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:30:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:30:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:30:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:30:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:30:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:30:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:30:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:30:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:30:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-13T09:30:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:30:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:30:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:30:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:30:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 8 2024-09-13T09:31:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:31:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:31:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:31:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:31:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:31:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:31:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:31:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:31:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:31:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:31:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:31:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:31:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:31:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:31:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:31:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:31:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:31:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:31:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:31:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:31:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:31:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:31:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:31:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:31:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:31:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:31:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:31:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:31:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:31:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:31:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:31:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:31:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:31:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:31:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:31:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:31:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:31:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:31:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:31:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:31:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:31:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:31:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:31:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:31:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:31:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:31:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:31:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:31:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:31:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:31:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:31:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:31:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:31:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:31:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:31:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:31:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:31:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:31:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:31:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:31:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:31:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:31:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:31:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:31:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:31:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:31:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:31:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:31:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:31:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:31:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:31:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:31:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:31:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:31:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:31:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:31:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:31:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:31:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:31:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:31:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:31:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:31:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:31:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:31:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:31:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:31:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:31:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:31:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:31:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:31:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:31:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:31:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:31:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:31:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:31:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:31:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:31:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:31:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:31:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:31:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:31:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:31:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:31:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:31:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:31:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:31:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:31:02.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:31:02.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:31:02.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:31:02.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:31:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:31:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:31:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:31:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:31:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-13T09:31:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:31:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:31:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:31:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:31:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:31:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:31:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:31:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:31:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:31:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:31:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:31:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:31:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:31:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:31:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:31:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:31:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:31:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:31:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:31:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:31:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:31:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:31:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:31:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:31:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:31:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:31:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:31:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:31:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:31:03.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:31:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:31:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:31:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:31:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-13T09:31:03.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:31:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:31:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:31:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:31:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:31:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:31:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:31:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:31:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-13T09:31:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:31:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:31:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:31:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:31:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-13T09:31:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:31:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:31:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-test.status 2024-09-13T09:31:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-test.status 2024-09-13T09:31:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:31:03.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:31:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-test.status 2024-09-13T09:31:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-test.status 2024-09-13T09:31:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:31:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:31:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-13T09:31:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:31:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-13T09:31:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 25 2024-09-13T09:31:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:31:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:31:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:31:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-13T09:31:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:31:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:31:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=25 2024-09-13T09:31:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:31:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:31:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:31:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:31:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:31:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:31:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:31:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:31:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:31:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:31:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:31:04.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:31:04.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:31:04.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:31:04.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 25 = '' 2024-09-13T09:31:04.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:31:04.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:31:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:31:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:31:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:31:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:31:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:31:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:31:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:31:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:31:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:31:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:31:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:31:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 25 = '' 2024-09-13T09:31:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:31:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:31:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:31:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:31:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:31:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:31:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:31:06.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:31:06.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:31:06.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:31:06.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:31:06.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:31:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:31:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 25 = '' 2024-09-13T09:31:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:31:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:31:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:31:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:31:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:31:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:31:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:31:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:31:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=25 2024-09-13T09:31:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 25 = 25 2024-09-13T09:31:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:31:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-09-13T09:31:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:31:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap6 2024-09-13T09:31:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:31:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:31:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:31:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-09-13T09:31:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2024-09-13T09:31:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T09:31:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:31:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 12 2024-09-13T09:31:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10ae4855b728 2024-09-13T09:31:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10ae4855b728 2024-09-13T09:31:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:31:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:31:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:31:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:31:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:31:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:28:45 2024 2024-09-13T09:31:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:31:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:31:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:31:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:31:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:31:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:31:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:31:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snap5-clean ']' 2024-09-13T09:31:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-13T09:31:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap5-clean 2024-09-13T09:31:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:31:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:31:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:31:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:31:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:31:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:31:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:31:10.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:31:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:31:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:31:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:31:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap6 2024-09-13T09:31:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap6 2024-09-13T09:31:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:31:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:31:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:31:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:31:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:31:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:31:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:31:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:31:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:31:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:31:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:31:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name=snap6-clean 2024-09-13T09:31:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-09-13T09:31:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:31:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap6-clean 2024-09-13T09:31:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-09-13T09:31:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:31:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror test 2024-09-13T09:31:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:31:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:31:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=7 2024-09-13T09:31:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:31:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-13T09:31:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 22761472 bytes 2024-09-13T09:31:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 49252352 bytes 2024-09-13T09:31:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 135392256 bytes 2024-09-13T09:31:13.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 147024896 bytes 2024-09-13T09:31:13.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 233257984 bytes 2024-09-13T09:31:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 257069056 bytes 2024-09-13T09:31:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 257697792 bytes 2024-09-13T09:31:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 269620224 bytes 2024-09-13T09:31:13.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 270258176 bytes 2024-09-13T09:31:13.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 296932352 bytes 2024-09-13T09:31:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 362451968 bytes 2024-09-13T09:31:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 402913280 bytes 2024-09-13T09:31:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 432687104 bytes 2024-09-13T09:31:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 468205568 bytes 2024-09-13T09:31:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 515432448 bytes 2024-09-13T09:31:14.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 565059584 bytes 2024-09-13T09:31:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 628644864 bytes 2024-09-13T09:31:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 678423552 bytes 2024-09-13T09:31:14.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 755896320 bytes 2024-09-13T09:31:14.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 855766016 bytes 2024-09-13T09:31:14.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 902162432 bytes 2024-09-13T09:31:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 910005248 bytes 2024-09-13T09:31:15.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1031387136 bytes 2024-09-13T09:31:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1079169024 bytes 2024-09-13T09:31:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1117537280 bytes 2024-09-13T09:31:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1135687680 bytes 2024-09-13T09:31:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1193571328 bytes 2024-09-13T09:31:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1261891584 bytes 2024-09-13T09:31:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1278564352 bytes 2024-09-13T09:31:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1326166016 bytes 2024-09-13T09:31:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1356871680 bytes 2024-09-13T09:31:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1397567488 bytes 2024-09-13T09:31:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1504871424 bytes 2024-09-13T09:31:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1568480256 bytes 2024-09-13T09:31:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1586929664 bytes 2024-09-13T09:31:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1617725440 bytes 2024-09-13T09:31:16.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1712826368 bytes 2024-09-13T09:31:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1802583040 bytes 2024-09-13T09:31:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1851717632 bytes 2024-09-13T09:31:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1883249664 bytes 2024-09-13T09:31:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1943734272 bytes 2024-09-13T09:31:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1993026560 bytes 2024-09-13T09:31:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 2031452160 bytes 2024-09-13T09:31:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2125534208 bytes 2024-09-13T09:31:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2157270016 bytes 2024-09-13T09:31:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2180659200 bytes 2024-09-13T09:31:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2288326656 bytes 2024-09-13T09:31:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2342360064 bytes 2024-09-13T09:31:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2389871616 bytes 2024-09-13T09:31:18.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2411466752 bytes 2024-09-13T09:31:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2457474048 bytes 2024-09-13T09:31:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2496928768 bytes 2024-09-13T09:31:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2580562944 bytes 2024-09-13T09:31:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2631183360 bytes 2024-09-13T09:31:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2672264192 bytes 2024-09-13T09:31:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2714085376 bytes 2024-09-13T09:31:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2773607424 bytes 2024-09-13T09:31:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:31:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:31:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:31:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:31:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-13T09:31:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:31:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:31:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:31:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:31:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:31:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:31:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-13T09:31:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:31:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:31:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:31:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:31:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:31:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:31:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-13T09:31:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='test: 2024-09-13T09:31:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:31:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+replaying 2024-09-13T09:31:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219774,"remote_snapshot_timestamp":1726219774,"replay_state":"idle"} 2024-09-13T09:31:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: service: mirror.3 on smithi190 2024-09-13T09:31:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13 09:31:01 2024-09-13T09:31:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2024-09-13T09:31:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2024-09-13T09:31:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+stopped 2024-09-13T09:31:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: local image is primary 2024-09-13T09:31:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13 09:31:01' 2024-09-13T09:31:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ state: .*up+replaying' 2024-09-13T09:31:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: state: up+replaying 2024-09-13T09:31:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ description: .*' 2024-09-13T09:31:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219774,"remote_snapshot_timestamp":1726219774,"replay_state":"idle"} 2024-09-13T09:31:20.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:31:20.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2024-09-13T09:31:20.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^up+' 2024-09-13T09:31:20.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2024-09-13T09:31:20.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep 'service: *mirror..* on ' 2024-09-13T09:31:20.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: service: mirror.3 on smithi190 2024-09-13T09:31:20.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update 2024-09-13T09:31:20.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-13T09:31:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update='2024-09-13 09:31:01' 2024-09-13T09:31:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-13 09:31:01' 2024-09-13T09:31:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:31:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:31:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local 'prev_last_update=2024-09-13 09:31:01' 2024-09-13T09:31:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:31:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:31:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-13T09:31:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='OKOKOK142214293mirror.3smithi19018.2.4-820-g48a7de7etrueOK43854388mirror.1smithi19018.2.4-820-g48a7de7efalseOK44274430mirror.0smithi19018.2.4-820-g48a7de7efalseOK44974500mirror.2smithi19018.2.4-820-g48a7de7efalseOKtest47e44e3e-e2e8-4660-b7fd-df61813d8d21up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219774,"remote_snapshot_timestamp":1726219774,"replay_state":"idle"}42214293mirror.3smithi1902024-09-13 09:31:01cluster2b8d40c1b-48a7-4e9b-8cbe-df65c34733d8up+stoppedlocal image is primary2024-09-13 09:31:01' 2024-09-13T09:31:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update state 2024-09-13T09:31:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-13T09:31:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update='2024-09-13 09:31:01' 2024-09-13T09:31:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-13T09:31:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=up+replaying 2024-09-13T09:31:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2024-09-13T09:31:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep up+replaying 2024-09-13T09:31:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2024-09-13T09:31:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:31:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:31:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap7 2024-09-13T09:31:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap7 2024-09-13T09:31:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:31:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap7 2024-09-13T09:31:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:31:41.376 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5489100745954629) > weight (0.25) 2024-09-13T09:31:41.377 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.35770241094286104) > weight (0.25) 2024-09-13T09:31:41.377 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7530150452157959) > weight (0.25) 2024-09-13T09:31:41.377 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9021102312067422) > weight (0.25) 2024-09-13T09:31:41.377 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2024-09-13T09:31:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-13T09:31:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:31:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-13T09:31:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:31:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:31:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:31:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:31:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:31:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-13T09:31:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:31:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:31:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:31:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:31:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-13T09:31:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:31:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:31:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:31:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:31:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:31:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:31:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:31:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:31:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:31:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:31:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:31:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:31:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:31:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:31:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:31:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:31:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:31:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:31:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:31:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:31:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:31:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:31:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:31:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:31:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:31:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:31:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:31:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:31:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:31:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:31:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:31:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:31:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:31:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:31:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:31:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:31:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:31:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:31:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:31:50.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:31:50.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:31:50.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:31:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:31:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:31:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:31:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:31:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:31:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:31:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:31:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:31:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:31:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:31:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:31:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:31:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:31:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:31:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:31:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:31:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:31:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:31:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:31:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:31:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:31:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:31:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:31:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:31:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:31:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:31:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:31:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:31:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:31:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:31:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:31:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:31:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:31:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:31:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:31:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:31:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:31:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:31:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:31:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:31:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:31:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:31:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:31:50.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:31:50.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:31:50.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:31:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:31:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:31:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:31:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:31:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:31:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:31:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:31:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:31:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:31:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:31:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:31:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:31:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:31:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:31:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:31:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:31:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:31:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:31:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:31:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:31:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:31:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:31:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:31:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:31:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:31:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:31:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:31:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:31:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:31:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:31:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:31:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:31:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:31:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:31:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:31:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:31:50.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:31:50.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:31:50.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:31:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:31:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-13T09:31:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:31:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:31:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:31:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:31:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:31:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:31:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:31:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:31:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:31:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:31:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:31:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:31:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:31:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:31:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:31:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:31:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:31:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:31:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:31:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:31:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:31:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:31:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:31:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:31:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:31:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:31:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:31:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:31:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:31:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:31:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:31:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:31:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:31:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-13T09:31:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:31:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:31:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:31:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:31:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:31:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:31:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:31:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:31:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-13T09:31:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:31:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:31:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:31:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:31:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-13T09:31:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:31:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:31:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-test.status 2024-09-13T09:31:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-test.status 2024-09-13T09:31:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:31:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:31:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-test.status 2024-09-13T09:31:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-test.status 2024-09-13T09:31:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:31:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:31:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-13T09:31:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:31:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-13T09:31:52.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 29 2024-09-13T09:31:52.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:31:52.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:31:52.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:52.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:52.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:31:52.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-13T09:31:52.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:31:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:31:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=29 2024-09-13T09:31:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:31:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:31:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:31:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:31:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:31:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:31:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:31:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:31:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:31:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:31:52.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:31:52.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:31:52.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:31:52.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:31:52.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 29 = '' 2024-09-13T09:31:52.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:31:52.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:31:52.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:31:52.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:31:52.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:52.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:52.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:31:52.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:31:52.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:31:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:31:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:31:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:31:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:31:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:31:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:31:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 29 = '' 2024-09-13T09:31:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:31:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:31:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:31:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:31:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:31:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:31:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:31:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:31:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:31:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:31:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:31:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:31:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:31:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 29 = '' 2024-09-13T09:31:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:31:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:31:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:31:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:31:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:31:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:31:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:31:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:31:55.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=29 2024-09-13T09:31:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 29 = 29 2024-09-13T09:31:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:31:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-09-13T09:31:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:31:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap7 2024-09-13T09:31:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:31:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:31:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:31:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-09-13T09:31:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2024-09-13T09:31:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T09:31:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:31:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 14 2024-09-13T09:31:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10ae4855b728 2024-09-13T09:31:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10ae4855b728 2024-09-13T09:31:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:31:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:31:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:31:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:31:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:31:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:31:09 2024 2024-09-13T09:31:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:31:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:31:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:31:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:31:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:31:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:31:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:31:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snap6-clean ']' 2024-09-13T09:31:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-13T09:31:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap6-clean 2024-09-13T09:31:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:31:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:31:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:31:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:31:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:31:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:31:57.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:31:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:31:57.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:31:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:31:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap7 2024-09-13T09:31:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:31:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:31:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap7 2024-09-13T09:31:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:31:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:31:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:31:58.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:31:58.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:31:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:31:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:31:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:31:59.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:32:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:32:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:32:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name=snap7-clean 2024-09-13T09:32:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-09-13T09:32:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:32:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap7-clean 2024-09-13T09:32:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-09-13T09:32:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:32:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror test 2024-09-13T09:32:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:32:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:32:03.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=11 2024-09-13T09:32:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:32:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-13T09:32:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 5489664 bytes 2024-09-13T09:32:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 27149312 bytes 2024-09-13T09:32:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 74597376 bytes 2024-09-13T09:32:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 78602240 bytes 2024-09-13T09:32:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 138810368 bytes 2024-09-13T09:32:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 172816384 bytes 2024-09-13T09:32:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 271199232 bytes 2024-09-13T09:32:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 306753536 bytes 2024-09-13T09:32:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 330993664 bytes 2024-09-13T09:32:04.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 353275904 bytes 2024-09-13T09:32:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 412767232 bytes 2024-09-13T09:32:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 473768960 bytes 2024-09-13T09:32:04.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 530202624 bytes 2024-09-13T09:32:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 538051584 bytes 2024-09-13T09:32:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 616582144 bytes 2024-09-13T09:32:04.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 694267904 bytes 2024-09-13T09:32:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 734749696 bytes 2024-09-13T09:32:05.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 805082112 bytes 2024-09-13T09:32:05.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 834250752 bytes 2024-09-13T09:32:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 865012736 bytes 2024-09-13T09:32:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 879294464 bytes 2024-09-13T09:32:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 919350272 bytes 2024-09-13T09:32:05.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 940419072 bytes 2024-09-13T09:32:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 941022208 bytes 2024-09-13T09:32:05.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 960009216 bytes 2024-09-13T09:32:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 960669696 bytes 2024-09-13T09:32:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1019960320 bytes 2024-09-13T09:32:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1083527168 bytes 2024-09-13T09:32:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1180467200 bytes 2024-09-13T09:32:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1201851392 bytes 2024-09-13T09:32:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1223140352 bytes 2024-09-13T09:32:06.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1307666432 bytes 2024-09-13T09:32:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1353965568 bytes 2024-09-13T09:32:06.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1399161856 bytes 2024-09-13T09:32:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1437263872 bytes 2024-09-13T09:32:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1480724480 bytes 2024-09-13T09:32:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1558170624 bytes 2024-09-13T09:32:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1577231360 bytes 2024-09-13T09:32:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1634891776 bytes 2024-09-13T09:32:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1705800704 bytes 2024-09-13T09:32:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1706430464 bytes 2024-09-13T09:32:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1730344960 bytes 2024-09-13T09:32:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1790527488 bytes 2024-09-13T09:32:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1853500416 bytes 2024-09-13T09:32:08.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1883204608 bytes 2024-09-13T09:32:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1904115712 bytes 2024-09-13T09:32:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1926351872 bytes 2024-09-13T09:32:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1985182720 bytes 2024-09-13T09:32:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1997591552 bytes 2024-09-13T09:32:08.477 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3591060022969501) > weight (0.25) 2024-09-13T09:32:08.477 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8462476226406591) > weight (0.25) 2024-09-13T09:32:08.477 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-13T09:32:08.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-13T09:32:08.478 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-13T09:32:08.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T09:32:08.477+0000 7fa254c85640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 40857) UID: 1000 2024-09-13T09:32:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2068462592 bytes 2024-09-13T09:32:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2090254336 bytes 2024-09-13T09:32:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2142278656 bytes 2024-09-13T09:32:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2148550656 bytes 2024-09-13T09:32:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2191610880 bytes 2024-09-13T09:32:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2306031616 bytes 2024-09-13T09:32:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2354164736 bytes 2024-09-13T09:32:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2372069376 bytes 2024-09-13T09:32:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2372676608 bytes 2024-09-13T09:32:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2396495872 bytes 2024-09-13T09:32:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2446152704 bytes 2024-09-13T09:32:09.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2530787328 bytes 2024-09-13T09:32:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2545308672 bytes 2024-09-13T09:32:09.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2563278848 bytes 2024-09-13T09:32:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2610197504 bytes 2024-09-13T09:32:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2629432320 bytes 2024-09-13T09:32:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2653654016 bytes 2024-09-13T09:32:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2673742848 bytes 2024-09-13T09:32:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2751845376 bytes 2024-09-13T09:32:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2789521408 bytes 2024-09-13T09:32:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2807230464 bytes 2024-09-13T09:32:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2838971392 bytes 2024-09-13T09:32:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2884982784 bytes 2024-09-13T09:32:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2899272704 bytes 2024-09-13T09:32:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 2937964544 bytes 2024-09-13T09:32:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 2957097984 bytes 2024-09-13T09:32:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 2995477504 bytes 2024-09-13T09:32:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3017144320 bytes 2024-09-13T09:32:11.479 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-13T09:32:11.479 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:32:11.480 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-13T09:32:11.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-13T09:32:11.481 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-13T09:32:11.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-13T09:32:11.485 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2024-09-13T09:32:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3046426624 bytes 2024-09-13T09:32:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3103193088 bytes 2024-09-13T09:32:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3137385472 bytes 2024-09-13T09:32:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3170128896 bytes 2024-09-13T09:32:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3229611008 bytes 2024-09-13T09:32:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3252966400 bytes 2024-09-13T09:32:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3253576704 bytes 2024-09-13T09:32:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3345121280 bytes 2024-09-13T09:32:12.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3362304000 bytes 2024-09-13T09:32:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3422911488 bytes 2024-09-13T09:32:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3477143552 bytes 2024-09-13T09:32:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3519925248 bytes 2024-09-13T09:32:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 3564790784 bytes 2024-09-13T09:32:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 3679272960 bytes 2024-09-13T09:32:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 3711220736 bytes 2024-09-13T09:32:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 3721874432 bytes 2024-09-13T09:32:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9400 IOs, 3741955072 bytes 2024-09-13T09:32:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9500 IOs, 3779450880 bytes 2024-09-13T09:32:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9600 IOs, 3820939264 bytes 2024-09-13T09:32:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9700 IOs, 3843563520 bytes 2024-09-13T09:32:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9800 IOs, 3891986432 bytes 2024-09-13T09:32:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9900 IOs, 3945145344 bytes 2024-09-13T09:32:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10000 IOs, 3979455488 bytes 2024-09-13T09:32:14.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:32:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:32:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:32:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:32:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-13T09:32:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:32:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:32:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:32:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:32:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:32:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:32:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-13T09:32:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:32:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:32:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:32:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:32:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:32:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:32:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-13T09:32:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='test: 2024-09-13T09:32:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:32:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+replaying 2024-09-13T09:32:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: replaying, {"bytes_per_second":17730150.4,"bytes_per_snapshot":534354944.0,"last_snapshot_bytes":531904512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219911,"remote_snapshot_timestamp":1726219911,"replay_state":"idle"} 2024-09-13T09:32:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: service: mirror.3 on smithi190 2024-09-13T09:32:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13 09:32:01 2024-09-13T09:32:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2024-09-13T09:32:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2024-09-13T09:32:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+stopped 2024-09-13T09:32:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: local image is primary 2024-09-13T09:32:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13 09:32:01' 2024-09-13T09:32:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ state: .*up+replaying' 2024-09-13T09:32:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: state: up+replaying 2024-09-13T09:32:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ description: .*' 2024-09-13T09:32:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: description: replaying, {"bytes_per_second":17730150.4,"bytes_per_snapshot":534354944.0,"last_snapshot_bytes":531904512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219911,"remote_snapshot_timestamp":1726219911,"replay_state":"idle"} 2024-09-13T09:32:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:32:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2024-09-13T09:32:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^up+' 2024-09-13T09:32:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2024-09-13T09:32:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep 'service: *mirror..* on ' 2024-09-13T09:32:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: service: mirror.3 on smithi190 2024-09-13T09:32:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update 2024-09-13T09:32:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-13T09:32:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update='2024-09-13 09:32:01' 2024-09-13T09:32:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-13 09:32:01' 2024-09-13T09:32:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:32:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:32:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local 'prev_last_update=2024-09-13 09:32:01' 2024-09-13T09:32:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:32:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:32:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-13T09:32:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='WARNINGWARNINGOK142214293mirror.3smithi19018.2.4-820-g48a7de7etrueOK43854388mirror.1smithi19018.2.4-820-g48a7de7efalseOK44274430mirror.0smithi19018.2.4-820-g48a7de7efalseOK4555mirror.2smithi19018.2.4-820-g48a7de7efalseWARNINGnot reporting statustest47e44e3e-e2e8-4660-b7fd-df61813d8d21up+replayingreplaying, {"bytes_per_second":17730150.4,"bytes_per_snapshot":534354944.0,"last_snapshot_bytes":531904512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219911,"remote_snapshot_timestamp":1726219911,"replay_state":"idle"}42214293mirror.3smithi1902024-09-13 09:32:01cluster2b8d40c1b-48a7-4e9b-8cbe-df65c34733d8up+stoppedlocal image is primary2024-09-13 09:32:01' 2024-09-13T09:32:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update state 2024-09-13T09:32:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-13T09:32:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update='2024-09-13 09:32:01' 2024-09-13T09:32:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-13T09:32:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=up+replaying 2024-09-13T09:32:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2024-09-13T09:32:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep up+replaying 2024-09-13T09:32:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2024-09-13T09:32:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:32:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:32:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap8 2024-09-13T09:32:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap8 2024-09-13T09:32:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:32:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap8 2024-09-13T09:32:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:32:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-13T09:32:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:32:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-13T09:32:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:32:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:32:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:32:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:32:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:32:45.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-13T09:32:45.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:32:45.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:45.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:45.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:32:45.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:32:45.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:32:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-13T09:32:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:32:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:32:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:32:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:32:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:32:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:32:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:32:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:32:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:32:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:32:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:32:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:32:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:32:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:32:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:32:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:32:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:32:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:32:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:32:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:32:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:32:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:32:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:32:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:32:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:32:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:32:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:32:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:32:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:32:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:32:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:32:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:32:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:32:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:32:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:32:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:32:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:32:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:32:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:32:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:32:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:32:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:32:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:32:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:32:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:32:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:32:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:32:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:32:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:32:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:32:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:32:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:32:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:32:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:32:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:32:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:32:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:32:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:32:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:32:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:32:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:32:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:32:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:32:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:32:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:32:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:32:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:32:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:32:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:32:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:32:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:32:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:32:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:32:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:32:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:32:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:32:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:32:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:32:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:32:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:32:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:32:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:32:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:32:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:32:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:32:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:32:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:32:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:32:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:32:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:32:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:32:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:32:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:32:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:32:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:32:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:32:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:32:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:32:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:32:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:32:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:32:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:32:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:32:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:32:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:32:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:32:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:32:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:32:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:32:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:32:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:32:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:32:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:32:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:32:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:32:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:32:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:32:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:32:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:32:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:32:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:32:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:32:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:32:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:32:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:32:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:32:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:32:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:32:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-13T09:32:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:32:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:32:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:32:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:32:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:32:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:32:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:32:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:32:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:32:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:32:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:32:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:32:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:32:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:32:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:32:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:32:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:32:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:32:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:32:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:32:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:32:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:32:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:32:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:32:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:32:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:32:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:32:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:32:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:32:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:32:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:32:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:32:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:32:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-13T09:32:45.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:32:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:32:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:32:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:32:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:32:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:32:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:32:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:32:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-13T09:32:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:32:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:32:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:32:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:32:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-13T09:32:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:32:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:32:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-test.status 2024-09-13T09:32:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-test.status 2024-09-13T09:32:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:32:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:32:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-test.status 2024-09-13T09:32:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-test.status 2024-09-13T09:32:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:32:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:32:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-13T09:32:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:32:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-13T09:32:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 33 2024-09-13T09:32:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:32:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:32:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:32:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-13T09:32:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:32:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:32:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=33 2024-09-13T09:32:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:32:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:32:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:32:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:32:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:32:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:32:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:32:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:32:47.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:32:47.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:32:47.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:32:47.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:32:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:32:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:32:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 33 = '' 2024-09-13T09:32:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:32:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:32:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:32:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:32:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:32:48.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:32:48.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:32:48.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:32:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:32:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:32:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:32:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:32:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:32:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 33 = '' 2024-09-13T09:32:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:32:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:32:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:32:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:32:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:32:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:32:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:32:49.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:32:49.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:32:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:32:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:32:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:32:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:32:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 33 = '' 2024-09-13T09:32:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:32:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:32:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:32:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:32:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:32:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:32:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:32:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:32:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=33 2024-09-13T09:32:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 33 = 33 2024-09-13T09:32:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:32:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-09-13T09:32:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:32:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap8 2024-09-13T09:32:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:32:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:32:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:32:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-09-13T09:32:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2024-09-13T09:32:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T09:32:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:32:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 16 2024-09-13T09:32:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10ae4855b728 2024-09-13T09:32:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10ae4855b728 2024-09-13T09:32:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:32:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:32:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:32:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:32:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:32:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:31:09 2024 2024-09-13T09:32:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:32:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:32:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:32:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:32:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:32:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:32:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:32:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snap7-clean ']' 2024-09-13T09:32:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-13T09:32:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap7-clean 2024-09-13T09:32:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:32:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:32:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:32:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:32:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:32:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:32:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:32:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:32:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:32:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:32:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:32:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap8 2024-09-13T09:32:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap8 2024-09-13T09:32:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:32:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:32:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:32:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:32:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:32:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:32:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:32:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:32:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:32:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:32:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:32:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name=snap8-clean 2024-09-13T09:32:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-09-13T09:32:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:32:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap8-clean 2024-09-13T09:32:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-09-13T09:32:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:32:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror test 2024-09-13T09:32:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:32:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:32:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:32:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:32:58.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=14 2024-09-13T09:32:58.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:32:58.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-13T09:32:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 47213568 bytes 2024-09-13T09:32:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 92419072 bytes 2024-09-13T09:32:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 117458944 bytes 2024-09-13T09:32:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 139734016 bytes 2024-09-13T09:32:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 193480704 bytes 2024-09-13T09:32:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 226515968 bytes 2024-09-13T09:32:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 233630720 bytes 2024-09-13T09:32:59.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 268095488 bytes 2024-09-13T09:32:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 272369664 bytes 2024-09-13T09:32:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 343280640 bytes 2024-09-13T09:32:59.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 362013696 bytes 2024-09-13T09:32:59.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 444385280 bytes 2024-09-13T09:32:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 477826048 bytes 2024-09-13T09:32:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 535539712 bytes 2024-09-13T09:32:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 551052288 bytes 2024-09-13T09:32:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 614059008 bytes 2024-09-13T09:33:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 639119360 bytes 2024-09-13T09:33:00.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 706578432 bytes 2024-09-13T09:33:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 773691392 bytes 2024-09-13T09:33:00.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 785538048 bytes 2024-09-13T09:33:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 875373568 bytes 2024-09-13T09:33:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 878579712 bytes 2024-09-13T09:33:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 910233600 bytes 2024-09-13T09:33:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 963846144 bytes 2024-09-13T09:33:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1014394880 bytes 2024-09-13T09:33:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1044811776 bytes 2024-09-13T09:33:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1106407424 bytes 2024-09-13T09:33:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1113975808 bytes 2024-09-13T09:33:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1169718272 bytes 2024-09-13T09:33:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1187726336 bytes 2024-09-13T09:33:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1209671680 bytes 2024-09-13T09:33:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1281651712 bytes 2024-09-13T09:33:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1307352064 bytes 2024-09-13T09:33:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1379463168 bytes 2024-09-13T09:33:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1420705792 bytes 2024-09-13T09:33:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1463739392 bytes 2024-09-13T09:33:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1488543744 bytes 2024-09-13T09:33:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1514210304 bytes 2024-09-13T09:33:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1553420288 bytes 2024-09-13T09:33:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1562945536 bytes 2024-09-13T09:33:02.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1636390912 bytes 2024-09-13T09:33:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1656188928 bytes 2024-09-13T09:33:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1695019008 bytes 2024-09-13T09:33:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1727978496 bytes 2024-09-13T09:33:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1741309952 bytes 2024-09-13T09:33:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1752627200 bytes 2024-09-13T09:33:03.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1806406656 bytes 2024-09-13T09:33:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1864065024 bytes 2024-09-13T09:33:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1877901312 bytes 2024-09-13T09:33:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1942517760 bytes 2024-09-13T09:33:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2013399040 bytes 2024-09-13T09:33:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2088411136 bytes 2024-09-13T09:33:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2131594240 bytes 2024-09-13T09:33:04.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2146792448 bytes 2024-09-13T09:33:04.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2184553472 bytes 2024-09-13T09:33:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2212043776 bytes 2024-09-13T09:33:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2237904896 bytes 2024-09-13T09:33:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2267909120 bytes 2024-09-13T09:33:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2303916032 bytes 2024-09-13T09:33:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2380933120 bytes 2024-09-13T09:33:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2385015808 bytes 2024-09-13T09:33:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2436123648 bytes 2024-09-13T09:33:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2497864704 bytes 2024-09-13T09:33:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2525390848 bytes 2024-09-13T09:33:05.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2562500608 bytes 2024-09-13T09:33:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2563113984 bytes 2024-09-13T09:33:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2600077312 bytes 2024-09-13T09:33:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2693672960 bytes 2024-09-13T09:33:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2696343552 bytes 2024-09-13T09:33:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2713197568 bytes 2024-09-13T09:33:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2728049664 bytes 2024-09-13T09:33:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2728896512 bytes 2024-09-13T09:33:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2748525568 bytes 2024-09-13T09:33:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 2773817344 bytes 2024-09-13T09:33:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 2797421568 bytes 2024-09-13T09:33:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 2847587328 bytes 2024-09-13T09:33:06.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 2854650880 bytes 2024-09-13T09:33:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 2880023552 bytes 2024-09-13T09:33:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 2923036672 bytes 2024-09-13T09:33:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3005282304 bytes 2024-09-13T09:33:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3076256768 bytes 2024-09-13T09:33:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3132161024 bytes 2024-09-13T09:33:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3222988800 bytes 2024-09-13T09:33:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3239095296 bytes 2024-09-13T09:33:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3312022528 bytes 2024-09-13T09:33:07.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3328580608 bytes 2024-09-13T09:33:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3361730560 bytes 2024-09-13T09:33:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3367287808 bytes 2024-09-13T09:33:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3484426240 bytes 2024-09-13T09:33:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 3518932992 bytes 2024-09-13T09:33:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 3519570944 bytes 2024-09-13T09:33:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 3562408960 bytes 2024-09-13T09:33:08.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 3582544896 bytes 2024-09-13T09:33:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9400 IOs, 3614201856 bytes 2024-09-13T09:33:08.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9500 IOs, 3671479296 bytes 2024-09-13T09:33:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9600 IOs, 3712922624 bytes 2024-09-13T09:33:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9700 IOs, 3755187200 bytes 2024-09-13T09:33:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9800 IOs, 3766468608 bytes 2024-09-13T09:33:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9900 IOs, 3838446592 bytes 2024-09-13T09:33:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10000 IOs, 3973388288 bytes 2024-09-13T09:33:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10100 IOs, 4008812544 bytes 2024-09-13T09:33:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10200 IOs, 4019690496 bytes 2024-09-13T09:33:09.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10300 IOs, 4020302848 bytes 2024-09-13T09:33:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10400 IOs, 4057362432 bytes 2024-09-13T09:33:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10500 IOs, 4074883072 bytes 2024-09-13T09:33:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10600 IOs, 4131717120 bytes 2024-09-13T09:33:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10700 IOs, 4145707008 bytes 2024-09-13T09:33:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10800 IOs, 4218466304 bytes 2024-09-13T09:33:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10900 IOs, 4235615232 bytes 2024-09-13T09:33:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11000 IOs, 4297464832 bytes 2024-09-13T09:33:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11100 IOs, 4299603968 bytes 2024-09-13T09:33:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11200 IOs, 4324397056 bytes 2024-09-13T09:33:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11300 IOs, 4367586304 bytes 2024-09-13T09:33:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11400 IOs, 4398253056 bytes 2024-09-13T09:33:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11500 IOs, 4438170624 bytes 2024-09-13T09:33:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11600 IOs, 4516319232 bytes 2024-09-13T09:33:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11700 IOs, 4588045312 bytes 2024-09-13T09:33:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11800 IOs, 4591199232 bytes 2024-09-13T09:33:11.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11900 IOs, 4628014080 bytes 2024-09-13T09:33:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12000 IOs, 4681897984 bytes 2024-09-13T09:33:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12100 IOs, 4761622528 bytes 2024-09-13T09:33:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12200 IOs, 4762204160 bytes 2024-09-13T09:33:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12300 IOs, 4853427200 bytes 2024-09-13T09:33:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12400 IOs, 4859857920 bytes 2024-09-13T09:33:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12500 IOs, 4927869952 bytes 2024-09-13T09:33:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:12600 IOs, 4955544576 bytes 2024-09-13T09:33:12.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:33:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:33:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:33:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:33:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-13T09:33:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:33:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:33:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:33:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:33:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:33:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:33:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:33:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:33:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-13T09:33:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:33:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:33:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:33:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:33:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:33:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:33:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:33:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:33:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-13T09:33:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='test: 2024-09-13T09:33:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:33:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+replaying 2024-09-13T09:33:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: replaying, {"bytes_per_second":17882453.33,"bytes_per_snapshot":534189056.0,"last_snapshot_bytes":536473600,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219966,"remote_snapshot_timestamp":1726219966,"replay_state":"idle"} 2024-09-13T09:33:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: service: mirror.3 on smithi190 2024-09-13T09:33:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13 09:33:01 2024-09-13T09:33:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2024-09-13T09:33:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2024-09-13T09:33:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+stopped 2024-09-13T09:33:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: local image is primary 2024-09-13T09:33:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13 09:33:01' 2024-09-13T09:33:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ state: .*up+replaying' 2024-09-13T09:33:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: state: up+replaying 2024-09-13T09:33:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ description: .*' 2024-09-13T09:33:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: description: replaying, {"bytes_per_second":17882453.33,"bytes_per_snapshot":534189056.0,"last_snapshot_bytes":536473600,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219966,"remote_snapshot_timestamp":1726219966,"replay_state":"idle"} 2024-09-13T09:33:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:33:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2024-09-13T09:33:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^up+' 2024-09-13T09:33:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2024-09-13T09:33:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep 'service: *mirror..* on ' 2024-09-13T09:33:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: service: mirror.3 on smithi190 2024-09-13T09:33:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update 2024-09-13T09:33:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-13T09:33:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update='2024-09-13 09:33:01' 2024-09-13T09:33:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-13 09:33:01' 2024-09-13T09:33:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:33:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:33:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:33:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:33:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local 'prev_last_update=2024-09-13 09:33:01' 2024-09-13T09:33:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:33:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:33:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-13T09:33:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='OKOKOK142214293mirror.3smithi19018.2.4-820-g48a7de7etrueOK43854388mirror.1smithi19018.2.4-820-g48a7de7efalseOK44274430mirror.0smithi19018.2.4-820-g48a7de7efalseOK45554558mirror.2smithi19018.2.4-820-g48a7de7efalseOKtest47e44e3e-e2e8-4660-b7fd-df61813d8d21up+replayingreplaying, {"bytes_per_second":17882453.33,"bytes_per_snapshot":534189056.0,"last_snapshot_bytes":536473600,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726219966,"remote_snapshot_timestamp":1726219966,"replay_state":"idle"}42214293mirror.3smithi1902024-09-13 09:33:01cluster2b8d40c1b-48a7-4e9b-8cbe-df65c34733d8up+stoppedlocal image is primary2024-09-13 09:33:01' 2024-09-13T09:33:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update state 2024-09-13T09:33:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-13T09:33:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update='2024-09-13 09:33:01' 2024-09-13T09:33:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-13T09:33:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=up+replaying 2024-09-13T09:33:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2024-09-13T09:33:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep up+replaying 2024-09-13T09:33:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2024-09-13T09:33:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:33:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:33:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap9 2024-09-13T09:33:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap9 2024-09-13T09:33:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:33:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:33:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:33:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap9 2024-09-13T09:33:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:33:34.486 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6519657279467018) > weight (0.25) 2024-09-13T09:33:34.486 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-13T09:33:34.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-13T09:33:34.487 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-13T09:33:34.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T09:33:34.487+0000 7f3e105d0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 39237) UID: 1000 2024-09-13T09:33:39.378 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-13T09:33:39.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-13T09:33:39.378 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-13T09:33:39.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:33:39.378+0000 7ff7f4f01640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 40750) UID: 1000 2024-09-13T09:33:40.488 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-13T09:33:40.488 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:33:40.488 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-13T09:33:40.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-13T09:33:40.489 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-13T09:33:40.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-13T09:33:40.492 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2024-09-13T09:33:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-13T09:33:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:33:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:33:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:33:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-13T09:33:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:33:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:33:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:33:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:33:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:33:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:33:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:33:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-13T09:33:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:33:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:33:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:33:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:33:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:33:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:33:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-13T09:33:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:33:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:33:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:33:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:33:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:33:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:33:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:33:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:33:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:33:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:33:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:33:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:33:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:33:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:33:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:33:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:33:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:33:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:33:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:33:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:33:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:33:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:33:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:33:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:33:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:33:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:33:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:33:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:33:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:33:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:33:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:33:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:33:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:33:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:33:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:33:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:33:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:33:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:33:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:33:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:33:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:33:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:33:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:33:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:33:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:33:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:33:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:33:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:33:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:33:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:33:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:33:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:33:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:33:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:33:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:33:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:33:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:33:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:33:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:33:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:33:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:33:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:33:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:33:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:33:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:33:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:33:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:33:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:33:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:33:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:33:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:33:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:33:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:33:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:33:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:33:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:33:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:33:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:33:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:33:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:33:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:33:43.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:33:43.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:33:43.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:33:43.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:33:43.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:33:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:33:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:33:43.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:33:43.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:33:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:33:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:33:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:33:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:33:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:33:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:33:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:33:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:33:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:33:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:33:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:33:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:33:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:33:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:33:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:33:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:33:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:33:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:33:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:33:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:33:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:33:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:33:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:33:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:33:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:33:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:33:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:33:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:33:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:33:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:33:43.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:33:43.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:33:43.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:33:43.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:33:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:33:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:33:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:33:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:33:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-13T09:33:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:33:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:33:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:33:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:33:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:33:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:33:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:33:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:33:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:33:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:33:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:33:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:33:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:33:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:33:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:33:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:33:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:33:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:33:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:33:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:33:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:33:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:33:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:33:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:33:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:33:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:33:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:33:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:33:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:33:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:33:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:33:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:33:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:33:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-13T09:33:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:33:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:33:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:33:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:33:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:33:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:33:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:33:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:33:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-13T09:33:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:33:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:33:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:33:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:33:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:33:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:33:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-13T09:33:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:33:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:33:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:33:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:33:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-test.status 2024-09-13T09:33:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-test.status 2024-09-13T09:33:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:33:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:33:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-test.status 2024-09-13T09:33:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-test.status 2024-09-13T09:33:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:33:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:33:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-13T09:33:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:33:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:33:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:33:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-13T09:33:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 37 2024-09-13T09:33:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:33:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:33:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:33:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:33:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:33:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:33:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-13T09:33:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:33:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=37 2024-09-13T09:33:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:33:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:33:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:33:45.379 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-13T09:33:45.380 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:33:45.380 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-13T09:33:45.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-13T09:33:45.380 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-13T09:33:45.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-13T09:33:45.383 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2024-09-13T09:33:45.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:33:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:33:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:33:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:33:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:33:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:33:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:33:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:33:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:33:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:33:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:33:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:33:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:33:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 37 = '' 2024-09-13T09:33:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:33:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:33:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:33:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:33:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:33:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:33:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:33:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:33:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:33:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:33:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:33:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:33:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:33:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:33:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:33:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 37 = '' 2024-09-13T09:33:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:33:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:33:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:33:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:33:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:33:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:33:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:33:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:33:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:33:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:33:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:33:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:33:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:33:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:33:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:33:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 37 = '' 2024-09-13T09:33:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:33:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:33:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:33:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:33:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:33:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:33:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:33:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:33:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:33:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:33:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=37 2024-09-13T09:33:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 37 = 37 2024-09-13T09:33:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:33:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-09-13T09:33:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:33:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:33:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:33:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap9 2024-09-13T09:33:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:33:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:33:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:33:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-09-13T09:33:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2024-09-13T09:33:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T09:33:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:33:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 18 2024-09-13T09:33:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10ae4855b728 2024-09-13T09:33:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10ae4855b728 2024-09-13T09:33:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:33:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:33:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:33:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:33:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:33:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:32:52 2024 2024-09-13T09:33:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:33:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:33:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:33:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:33:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:33:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:33:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:33:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snap8-clean ']' 2024-09-13T09:33:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-13T09:33:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:33:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:33:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap8-clean 2024-09-13T09:33:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:33:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:33:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:33:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:33:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:33:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:33:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:33:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:33:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:33:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:33:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:33:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap9 2024-09-13T09:33:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:33:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:33:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap9 2024-09-13T09:33:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:33:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:33:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:33:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:33:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:33:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:33:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:33:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:33:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:33:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:33:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:33:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name=snap9-clean 2024-09-13T09:33:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-09-13T09:33:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:33:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:33:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:33:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap9-clean 2024-09-13T09:33:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-09-13T09:33:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:33:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror test 2024-09-13T09:33:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:33:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:33:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:33:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:33:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=13 2024-09-13T09:33:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:33:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-13T09:33:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 82216960 bytes 2024-09-13T09:33:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 163474432 bytes 2024-09-13T09:33:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 197007360 bytes 2024-09-13T09:33:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 267937792 bytes 2024-09-13T09:33:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 331039744 bytes 2024-09-13T09:33:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 385054720 bytes 2024-09-13T09:33:54.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 469812224 bytes 2024-09-13T09:33:54.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 513570816 bytes 2024-09-13T09:33:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 538556416 bytes 2024-09-13T09:33:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 577941504 bytes 2024-09-13T09:33:54.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 638157824 bytes 2024-09-13T09:33:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 762122240 bytes 2024-09-13T09:33:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 789658624 bytes 2024-09-13T09:33:55.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 821242880 bytes 2024-09-13T09:33:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 885269504 bytes 2024-09-13T09:33:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 931843072 bytes 2024-09-13T09:33:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 967064576 bytes 2024-09-13T09:33:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 1002728448 bytes 2024-09-13T09:33:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 1023714304 bytes 2024-09-13T09:33:56.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 1064693760 bytes 2024-09-13T09:33:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1113674752 bytes 2024-09-13T09:33:56.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1123855360 bytes 2024-09-13T09:33:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1217948672 bytes 2024-09-13T09:33:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1253186560 bytes 2024-09-13T09:33:56.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1293188096 bytes 2024-09-13T09:33:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1315093504 bytes 2024-09-13T09:33:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1335695360 bytes 2024-09-13T09:33:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1391746048 bytes 2024-09-13T09:33:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1412456448 bytes 2024-09-13T09:33:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1440580608 bytes 2024-09-13T09:33:57.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1470897152 bytes 2024-09-13T09:33:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1494712320 bytes 2024-09-13T09:33:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1519108096 bytes 2024-09-13T09:33:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1536634880 bytes 2024-09-13T09:33:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1585624064 bytes 2024-09-13T09:33:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1638794240 bytes 2024-09-13T09:33:57.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1656702976 bytes 2024-09-13T09:33:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1667185664 bytes 2024-09-13T09:33:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1692673024 bytes 2024-09-13T09:33:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1714700288 bytes 2024-09-13T09:33:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1733183488 bytes 2024-09-13T09:33:58.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1769102336 bytes 2024-09-13T09:33:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1851278336 bytes 2024-09-13T09:33:58.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1895845888 bytes 2024-09-13T09:33:58.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1896466432 bytes 2024-09-13T09:33:58.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1922117632 bytes 2024-09-13T09:33:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1955654656 bytes 2024-09-13T09:33:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1974765568 bytes 2024-09-13T09:33:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1997130752 bytes 2024-09-13T09:33:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2030323712 bytes 2024-09-13T09:33:59.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2086667264 bytes 2024-09-13T09:33:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2097017856 bytes 2024-09-13T09:33:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2135790592 bytes 2024-09-13T09:33:59.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2185723904 bytes 2024-09-13T09:33:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2237028352 bytes 2024-09-13T09:33:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2283793408 bytes 2024-09-13T09:33:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2351923200 bytes 2024-09-13T09:34:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2362128384 bytes 2024-09-13T09:34:00.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2397410304 bytes 2024-09-13T09:34:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2412495872 bytes 2024-09-13T09:34:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2453575680 bytes 2024-09-13T09:34:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2475795456 bytes 2024-09-13T09:34:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2495812608 bytes 2024-09-13T09:34:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2544451584 bytes 2024-09-13T09:34:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2606967808 bytes 2024-09-13T09:34:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2633013248 bytes 2024-09-13T09:34:01.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2684276736 bytes 2024-09-13T09:34:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2715632640 bytes 2024-09-13T09:34:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2790031360 bytes 2024-09-13T09:34:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2813230080 bytes 2024-09-13T09:34:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2836970496 bytes 2024-09-13T09:34:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2849901568 bytes 2024-09-13T09:34:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2905193472 bytes 2024-09-13T09:34:01.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 2998924288 bytes 2024-09-13T09:34:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3053914112 bytes 2024-09-13T09:34:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3082237952 bytes 2024-09-13T09:34:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3106895872 bytes 2024-09-13T09:34:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3168845824 bytes 2024-09-13T09:34:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3193280512 bytes 2024-09-13T09:34:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3227034624 bytes 2024-09-13T09:34:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3306011648 bytes 2024-09-13T09:34:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3314991104 bytes 2024-09-13T09:34:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3324080128 bytes 2024-09-13T09:34:02.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3432408064 bytes 2024-09-13T09:34:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3458904064 bytes 2024-09-13T09:34:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3509800960 bytes 2024-09-13T09:34:03.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3578444800 bytes 2024-09-13T09:34:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3616291840 bytes 2024-09-13T09:34:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3674030080 bytes 2024-09-13T09:34:03.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 3711117312 bytes 2024-09-13T09:34:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 3718243328 bytes 2024-09-13T09:34:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 3787350016 bytes 2024-09-13T09:34:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 3830505472 bytes 2024-09-13T09:34:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9400 IOs, 3854713856 bytes 2024-09-13T09:34:04.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9500 IOs, 3916310528 bytes 2024-09-13T09:34:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9600 IOs, 3951187968 bytes 2024-09-13T09:34:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9700 IOs, 3985552384 bytes 2024-09-13T09:34:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9800 IOs, 4077996032 bytes 2024-09-13T09:34:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9900 IOs, 4104587264 bytes 2024-09-13T09:34:04.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10000 IOs, 4140438528 bytes 2024-09-13T09:34:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10100 IOs, 4253093888 bytes 2024-09-13T09:34:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10200 IOs, 4282652672 bytes 2024-09-13T09:34:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10300 IOs, 4297273344 bytes 2024-09-13T09:34:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10400 IOs, 4362352640 bytes 2024-09-13T09:34:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10500 IOs, 4396106752 bytes 2024-09-13T09:34:05.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10600 IOs, 4467972096 bytes 2024-09-13T09:34:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10700 IOs, 4532212736 bytes 2024-09-13T09:34:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10800 IOs, 4607758336 bytes 2024-09-13T09:34:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10900 IOs, 4690966528 bytes 2024-09-13T09:34:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11000 IOs, 4765691904 bytes 2024-09-13T09:34:06.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11100 IOs, 4805983232 bytes 2024-09-13T09:34:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:34:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:34:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:34:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:34:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-13T09:34:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:34:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:34:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:34:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:34:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:34:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:34:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-13T09:34:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:34:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:34:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local description_pattern= 2024-09-13T09:34:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local service_pattern= 2024-09-13T09:34:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:34:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:34:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-13T09:34:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='test: 2024-09-13T09:34:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: global_id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:34:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+replaying 2024-09-13T09:34:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: replaying, {"bytes_per_second":17894604.8,"bytes_per_snapshot":536655872.0,"last_snapshot_bytes":536838144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726220024,"remote_snapshot_timestamp":1726220024,"replay_state":"idle"} 2024-09-13T09:34:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: service: mirror.3 on smithi190 2024-09-13T09:34:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13 09:34:01 2024-09-13T09:34:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: peer_sites: 2024-09-13T09:34:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: name: cluster2 2024-09-13T09:34:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: state: up+stopped 2024-09-13T09:34:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: description: local image is primary 2024-09-13T09:34:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: last_update: 2024-09-13 09:34:01' 2024-09-13T09:34:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ state: .*up+replaying' 2024-09-13T09:34:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: state: up+replaying 2024-09-13T09:34:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^ description: .*' 2024-09-13T09:34:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: description: replaying, {"bytes_per_second":17894604.8,"bytes_per_snapshot":536655872.0,"last_snapshot_bytes":536838144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726220024,"remote_snapshot_timestamp":1726220024,"replay_state":"idle"} 2024-09-13T09:34:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:34:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2024-09-13T09:34:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep '^up+' 2024-09-13T09:34:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2024-09-13T09:34:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep 'service: *mirror..* on ' 2024-09-13T09:34:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: service: mirror.3 on smithi190 2024-09-13T09:34:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update 2024-09-13T09:34:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-13T09:34:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update='2024-09-13 09:34:01' 2024-09-13T09:34:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-13 09:34:01' 2024-09-13T09:34:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:34:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state_pattern=up+replaying 2024-09-13T09:34:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local 'prev_last_update=2024-09-13 09:34:01' 2024-09-13T09:34:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status 2024-09-13T09:34:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ CEPH_ARGS= 2024-09-13T09:34:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-13T09:34:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status='OKOKOK142214293mirror.3smithi19018.2.4-820-g48a7de7etrueOK44274430mirror.0smithi19018.2.4-820-g48a7de7efalseOK45554558mirror.2smithi19018.2.4-820-g48a7de7efalseOK45984601mirror.1smithi19018.2.4-820-g48a7de7efalseOKtest47e44e3e-e2e8-4660-b7fd-df61813d8d21up+replayingreplaying, {"bytes_per_second":17894604.8,"bytes_per_snapshot":536655872.0,"last_snapshot_bytes":536838144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726220024,"remote_snapshot_timestamp":1726220024,"replay_state":"idle"}42214293mirror.3smithi1902024-09-13 09:34:01cluster2b8d40c1b-48a7-4e9b-8cbe-df65c34733d8up+stoppedlocal image is primary2024-09-13 09:34:01' 2024-09-13T09:34:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_update state 2024-09-13T09:34:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-13T09:34:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ last_update='2024-09-13 09:34:01' 2024-09-13T09:34:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-13T09:34:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=up+replaying 2024-09-13T09:34:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo up+replaying 2024-09-13T09:34:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep up+replaying 2024-09-13T09:34:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:up+replaying 2024-09-13T09:34:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:34:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:34:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap10 2024-09-13T09:34:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap10 2024-09-13T09:34:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:34:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap10 2024-09-13T09:34:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:34:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-13T09:34:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:34:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-13T09:34:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:34:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:34:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:34:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:34:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:34:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-13T09:34:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:34:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:34:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:34:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:34:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-13T09:34:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:34:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:34:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:34:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:34:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:34:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:34:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:34:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:34:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:34:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:34:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:34:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:34:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:34:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:34:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:34:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:34:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:34:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:34:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:34:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:34:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:34:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:34:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:34:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:34:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:34:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:34:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:34:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:34:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:34:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:34:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:34:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:34:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:34:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:34:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:34:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:34:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:34:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:34:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:34:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:34:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:34:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:34:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:34:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:34:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:34:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:34:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:34:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:34:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:34:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:34:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:34:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:34:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:34:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:34:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:34:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:34:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:34:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:34:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:34:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:34:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:34:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:34:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:34:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:34:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:34:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:34:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:34:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:34:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:34:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:34:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:34:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:34:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:34:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:34:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:34:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:34:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:34:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:34:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:34:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:34:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:34:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:34:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:34:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:34:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:34:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:34:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:34:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:34:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:34:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:34:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:34:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:34:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:34:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:34:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:34:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:34:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:34:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:34:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:34:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:34:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:34:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:34:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:34:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:34:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:34:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:34:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:34:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:34:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:34:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:34:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:34:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:34:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:34:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:34:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:34:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:34:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:34:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:34:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:34:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:34:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:34:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:34:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:34:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:34:37.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:34:37.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:34:37.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:34:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:34:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-13T09:34:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:34:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:34:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:34:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:34:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:34:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:34:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:34:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:34:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:34:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:34:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:34:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:34:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:34:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:34:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:34:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:34:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:34:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:34:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:34:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:34:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:34:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:34:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:34:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:34:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:34:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:34:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:34:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:34:37.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:34:37.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:34:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:34:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:34:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:34:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-13T09:34:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:34:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:34:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:34:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:34:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:34:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:34:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:34:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:34:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-13T09:34:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:34:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:34:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:34:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:34:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-13T09:34:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:34:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:34:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-test.status 2024-09-13T09:34:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-test.status 2024-09-13T09:34:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:34:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:34:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-test.status 2024-09-13T09:34:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-test.status 2024-09-13T09:34:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:34:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:34:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-13T09:34:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:34:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-13T09:34:39.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 41 2024-09-13T09:34:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:34:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:34:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:34:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:34:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-13T09:34:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:34:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=41 2024-09-13T09:34:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:34:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:34:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:34:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:34:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:34:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:34:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:34:39.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:34:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:34:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:34:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:34:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:34:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:34:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:34:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 41 = '' 2024-09-13T09:34:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:34:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:34:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:34:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:34:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:34:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:34:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:34:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:34:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:34:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:34:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:34:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:34:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:34:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 41 = '' 2024-09-13T09:34:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:34:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:34:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:34:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:34:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:34:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:34:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:34:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:34:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:34:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:34:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:34:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:34:41.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:34:41.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 41 = '' 2024-09-13T09:34:41.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:34:41.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:34:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:34:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:34:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:34:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:34:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:34:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:34:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=41 2024-09-13T09:34:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 41 = 41 2024-09-13T09:34:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:34:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-09-13T09:34:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:34:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap10 2024-09-13T09:34:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:34:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:34:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:34:43.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-09-13T09:34:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2024-09-13T09:34:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T09:34:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:34:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 20 2024-09-13T09:34:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10ae4855b728 2024-09-13T09:34:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10ae4855b728 2024-09-13T09:34:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:34:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:34:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:34:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:34:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:34:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:32:52 2024 2024-09-13T09:34:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:34:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:34:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:34:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:34:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:34:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:34:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:34:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snap9-clean ']' 2024-09-13T09:34:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-13T09:34:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap9-clean 2024-09-13T09:34:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:34:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:34:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:34:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:34:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:45.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:34:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:34:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap10 2024-09-13T09:34:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap10 2024-09-13T09:34:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:34:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:34:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:34:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:34:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:46.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:34:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:34:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ clean_snap_name=snap10-clean 2024-09-13T09:34:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-09-13T09:34:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:34:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap10-clean 2024-09-13T09:34:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-09-13T09:34:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-13T09:34:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:34:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-13T09:34:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:34:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:34:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:34:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:34:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:34:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-13T09:34:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:34:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:34:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:34:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:34:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-13T09:34:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:34:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:34:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:34:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:34:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:34:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:34:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:34:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:34:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:34:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:34:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:34:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-13T09:34:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:34:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:34:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:34:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:34:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:34:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:34:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:34:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:34:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:34:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:34:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:34:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:34:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:34:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:34:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:34:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:34:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:34:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:34:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:34:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:34:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:34:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:34:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:34:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:34:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:34:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:34:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:34:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:34:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:34:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:34:49.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:34:49.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:34:49.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:34:49.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-13T09:34:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:34:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:34:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:34:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:34:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-13T09:34:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:34:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:34:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:34:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:34:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:34:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:34:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:34:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:34:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:34:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:34:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:34:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:34:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:34:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:34:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:34:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:34:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:34:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:34:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:34:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:34:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:34:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:34:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:34:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:34:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:34:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:34:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:34:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:34:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:34:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:34:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:34:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:34:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:34:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-13T09:34:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:34:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:34:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:34:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:34:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-13T09:34:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:34:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:34:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:34:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:34:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:34:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:34:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:34:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:34:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:34:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:34:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:34:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:34:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:34:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:34:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:34:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:34:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:34:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:34:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:34:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:34:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:34:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:34:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:34:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:34:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:34:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:34:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:34:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:34:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:34:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:34:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:34:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:34:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:34:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-13T09:34:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-13T09:34:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:34:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:34:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:34:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-13T09:34:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:34:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:34:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:34:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:34:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:34:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:34:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:34:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:34:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:34:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:34:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:34:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:34:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:34:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:34:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:34:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:34:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:34:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:34:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:34:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:34:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:34:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:34:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:34:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:34:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:34:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:34:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:34:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:34:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:34:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:34:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:34:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:34:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:34:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-13T09:34:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:34:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:34:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:34:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:34:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:34:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:34:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:34:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:34:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-13T09:34:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:34:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:34:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:34:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:34:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-13T09:34:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:34:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:34:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-test.status 2024-09-13T09:34:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-test.status 2024-09-13T09:34:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:34:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:34:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-test.status 2024-09-13T09:34:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-test.status 2024-09-13T09:34:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:34:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:34:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-13T09:34:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:34:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-13T09:34:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 44 2024-09-13T09:34:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:34:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:34:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.status 2024-09-13T09:34:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-13T09:34:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:34:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:34:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=44 2024-09-13T09:34:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:34:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:34:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:34:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:34:51.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:34:51.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:51.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:51.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:34:51.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:34:51.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:34:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:34:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:34:51.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:34:51.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:34:51.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:34:51.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:34:51.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 44 = '' 2024-09-13T09:34:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:34:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:34:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:34:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:34:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.status 2024-09-13T09:34:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-13T09:34:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:34:51.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:34:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=44 2024-09-13T09:34:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 44 = 44 2024-09-13T09:34:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:34:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-09-13T09:34:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:34:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap10-clean 2024-09-13T09:34:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:34:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:34:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:34:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-09-13T09:34:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'test': 2024-09-13T09:34:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T09:34:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:34:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 21 2024-09-13T09:34:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10ae4855b728 2024-09-13T09:34:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10ae4855b728 2024-09-13T09:34:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:34:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:34:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:34:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:34:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:34:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:34:44 2024 2024-09-13T09:34:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:25:31 2024 2024-09-13T09:34:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:34:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:34:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:34:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 47e44e3e-e2e8-4660-b7fd-df61813d8d21 2024-09-13T09:34:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:34:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:34:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ seq 1 10 2024-09-13T09:34:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:34:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap1 2024-09-13T09:34:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap1 2024-09-13T09:34:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap1 2024-09-13T09:34:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:34:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:34:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:34:53.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:34:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:34:54.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:34:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap1-clean 2024-09-13T09:34:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-13T09:34:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap1-clean 2024-09-13T09:34:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:34:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:34:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:34:55.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:34:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:34:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:34:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:34:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap2 2024-09-13T09:34:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap2 2024-09-13T09:34:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap2 2024-09-13T09:34:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:34:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:34:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:34:56.384 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-13T09:34:56.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-13T09:34:56.385 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-13T09:34:56.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:34:56.384+0000 7f529b5f3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 42569) UID: 1000 2024-09-13T09:34:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:34:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:34:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:34:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap2-clean 2024-09-13T09:34:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-13T09:34:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap2-clean 2024-09-13T09:34:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:34:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:34:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:34:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:34:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:34:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:34:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:34:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap3 2024-09-13T09:34:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap3 2024-09-13T09:34:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:34:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:34:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap3 2024-09-13T09:34:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:34:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:34:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:34:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:34:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:34:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:00.386 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-13T09:35:00.386 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:35:00.386 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-13T09:35:00.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-13T09:35:00.387 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-13T09:35:00.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-13T09:35:00.391 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2024-09-13T09:35:00.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:00.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:35:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap3-clean 2024-09-13T09:35:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-13T09:35:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap3-clean 2024-09-13T09:35:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:35:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:02.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:35:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:35:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap4 2024-09-13T09:35:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap4 2024-09-13T09:35:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap4 2024-09-13T09:35:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:35:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:03.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:03.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:03.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:35:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap4-clean 2024-09-13T09:35:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-13T09:35:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap4-clean 2024-09-13T09:35:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:35:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:04.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:05.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:35:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:35:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap5 2024-09-13T09:35:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap5 2024-09-13T09:35:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap5 2024-09-13T09:35:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:35:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:07.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:07.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:35:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap5-clean 2024-09-13T09:35:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-13T09:35:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap5-clean 2024-09-13T09:35:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:35:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:35:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:35:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap6 2024-09-13T09:35:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap6 2024-09-13T09:35:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap6 2024-09-13T09:35:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:35:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:09.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:35:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap6-clean 2024-09-13T09:35:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-13T09:35:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap6-clean 2024-09-13T09:35:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:35:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:11.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:12.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:35:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:35:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap7 2024-09-13T09:35:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap7 2024-09-13T09:35:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap7 2024-09-13T09:35:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:35:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:35:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap7-clean 2024-09-13T09:35:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-13T09:35:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap7-clean 2024-09-13T09:35:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:35:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:17.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:35:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:35:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap8 2024-09-13T09:35:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap8 2024-09-13T09:35:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap8 2024-09-13T09:35:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:35:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:35:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap8-clean 2024-09-13T09:35:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-13T09:35:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap8-clean 2024-09-13T09:35:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:35:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:23.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:35:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:35:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap9 2024-09-13T09:35:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap9 2024-09-13T09:35:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap9 2024-09-13T09:35:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:35:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:24.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:25.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:35:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap9-clean 2024-09-13T09:35:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-13T09:35:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap9-clean 2024-09-13T09:35:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:35:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:26.493 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.787474128542799) > weight (0.25) 2024-09-13T09:35:26.493 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5660206097683446) > weight (0.25) 2024-09-13T09:35:26.493 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6609377670442617) > weight (0.25) 2024-09-13T09:35:26.493 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.33155560839388676) > weight (0.25) 2024-09-13T09:35:26.493 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2024-09-13T09:35:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:35:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:35:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap10 2024-09-13T09:35:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap10 2024-09-13T09:35:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap10 2024-09-13T09:35:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:35:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:35:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap10-clean 2024-09-13T09:35:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror test snap10-clean 2024-09-13T09:35:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap10-clean 2024-09-13T09:35:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:35:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export 2024-09-13T09:35:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T09:35:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-test.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-test.export 2024-09-13T09:35:30.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:35:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ seq 1 10 2024-09-13T09:35:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:35:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap1 2024-09-13T09:35:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-09-13T09:35:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap1 2024-09-13T09:35:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-09-13T09:35:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:35:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap1-clean 2024-09-13T09:35:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-09-13T09:35:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap1-clean 2024-09-13T09:35:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-09-13T09:35:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:35:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:35:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap2 2024-09-13T09:35:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-09-13T09:35:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap2 2024-09-13T09:35:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-09-13T09:35:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:35:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap2-clean 2024-09-13T09:35:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-09-13T09:35:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap2-clean 2024-09-13T09:35:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-09-13T09:35:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:35:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:35:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap3 2024-09-13T09:35:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-09-13T09:35:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap3 2024-09-13T09:35:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-09-13T09:35:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:35:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap3-clean 2024-09-13T09:35:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-09-13T09:35:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap3-clean 2024-09-13T09:35:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-09-13T09:35:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:35:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:35:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap4 2024-09-13T09:35:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-09-13T09:35:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap4 2024-09-13T09:35:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-09-13T09:35:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:35:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap4-clean 2024-09-13T09:35:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-09-13T09:35:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap4-clean 2024-09-13T09:35:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-09-13T09:35:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:35:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:35:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap5 2024-09-13T09:35:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-09-13T09:35:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap5 2024-09-13T09:35:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-09-13T09:35:39.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:35:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap5-clean 2024-09-13T09:35:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-09-13T09:35:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap5-clean 2024-09-13T09:35:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-09-13T09:35:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:35:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:35:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap6 2024-09-13T09:35:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-09-13T09:35:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap6 2024-09-13T09:35:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-09-13T09:35:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:35:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap6-clean 2024-09-13T09:35:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-09-13T09:35:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap6-clean 2024-09-13T09:35:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-09-13T09:35:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:35:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:35:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap7 2024-09-13T09:35:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-09-13T09:35:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap7 2024-09-13T09:35:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-09-13T09:35:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:35:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap7-clean 2024-09-13T09:35:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-09-13T09:35:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap7-clean 2024-09-13T09:35:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-09-13T09:35:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:35:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:35:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap8 2024-09-13T09:35:44.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-09-13T09:35:44.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:44.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:44.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:44.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap8 2024-09-13T09:35:44.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-09-13T09:35:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:35:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap8-clean 2024-09-13T09:35:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-09-13T09:35:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap8-clean 2024-09-13T09:35:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-09-13T09:35:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:35:46.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:35:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap9 2024-09-13T09:35:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-09-13T09:35:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap9 2024-09-13T09:35:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-09-13T09:35:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:35:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap9-clean 2024-09-13T09:35:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-09-13T09:35:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap9-clean 2024-09-13T09:35:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-09-13T09:35:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:35:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:35:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap10 2024-09-13T09:35:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-09-13T09:35:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap10 2024-09-13T09:35:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-09-13T09:35:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:35:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap10-clean 2024-09-13T09:35:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-09-13T09:35:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap10-clean 2024-09-13T09:35:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-09-13T09:35:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:35:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror test 2024-09-13T09:35:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:35:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:35:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror test 2024-09-13T09:35:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-09-13T09:35:50.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-09-13T09:35:50.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:35:50.207+0000 7f9a4a2d2640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b19573b520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:35:50.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:35:50.207+0000 7f9a50adf640 -1 rbd::mirror::ImageReplayer: 0x55b194972280 [3/47e44e3e-e2e8-4660-b7fd-df61813d8d21] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:35:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-13T09:35:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:35:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-09-13T09:35:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:35:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:35:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:35:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:35:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:35:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror test 2024-09-13T09:35:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:35:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:35:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=test 2024-09-13T09:35:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-13T09:35:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:35:51.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-13T09:35:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:35:51.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:35:51.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:35:51.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-09-13T09:35:51.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:35:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=test 2024-09-13T09:35:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:35:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:35:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:35:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:35:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror test 2024-09-13T09:35:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:35:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:35:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=test 2024-09-13T09:35:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-13T09:35:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:35:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-13T09:35:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:35:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:35:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:35:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:35:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ testlog 'TEST: create many images' 2024-09-13T09:35:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee -a /tmp/tmp.V7YewHEkvl/rbd-mirror.test.log 2024-09-13T09:35:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ date '+%F %T' 2024-09-13T09:35:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo 2024-09-13 09:35:51 TEST: create many images 2024-09-13T09:35:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:2024-09-13 09:35:51 TEST: create many images 2024-09-13T09:35:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ snap_name=snap 2024-09-13T09:35:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ seq 1 50 2024-09-13T09:35:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:35:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_1 2024-09-13T09:35:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2024-09-13T09:35:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:35:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:35:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:35:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:35:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:35:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:35:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:35:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_1 128M 2024-09-13T09:35:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:35:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:35:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:35:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:35:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:35:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:35:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:35:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:35:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_1 2024-09-13T09:35:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:35:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2024-09-13T09:35:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:35:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:35:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-13T09:35:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:35:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-13T09:35:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_1': 2024-09-13T09:35:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:35:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:35:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:35:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1278f6e70745 2024-09-13T09:35:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1278f6e70745 2024-09-13T09:35:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:35:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:35:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:35:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:35:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:35:51 2024 2024-09-13T09:35:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:35:51 2024 2024-09-13T09:35:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:35:51 2024 2024-09-13T09:35:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:35:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:35:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 6c1c05c0-9a83-4822-98dd-9fa3a496df28 2024-09-13T09:35:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:35:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:35:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-13T09:35:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:35:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:35:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:35:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:35:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2024-09-13T09:35:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:35:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:35:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 113250304 bytes 2024-09-13T09:35:52.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 144915456 bytes 2024-09-13T09:35:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 165638144 bytes 2024-09-13T09:35:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 224271360 bytes 2024-09-13T09:35:52.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 288748544 bytes 2024-09-13T09:35:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 338260992 bytes 2024-09-13T09:35:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 354041856 bytes 2024-09-13T09:35:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 434000896 bytes 2024-09-13T09:35:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 462898176 bytes 2024-09-13T09:35:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 478902272 bytes 2024-09-13T09:35:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 528900096 bytes 2024-09-13T09:35:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 537755648 bytes 2024-09-13T09:35:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 597323776 bytes 2024-09-13T09:35:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 659103744 bytes 2024-09-13T09:35:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 677188608 bytes 2024-09-13T09:35:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 709755904 bytes 2024-09-13T09:35:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 726040576 bytes 2024-09-13T09:35:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 811459584 bytes 2024-09-13T09:35:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 880306176 bytes 2024-09-13T09:35:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 964261888 bytes 2024-09-13T09:35:54.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1000443904 bytes 2024-09-13T09:35:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1043646464 bytes 2024-09-13T09:35:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1045145600 bytes 2024-09-13T09:35:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1157118976 bytes 2024-09-13T09:35:55.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1234117632 bytes 2024-09-13T09:35:55.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1297108992 bytes 2024-09-13T09:35:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1320192000 bytes 2024-09-13T09:35:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1387154432 bytes 2024-09-13T09:35:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1417090048 bytes 2024-09-13T09:35:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1457557504 bytes 2024-09-13T09:35:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1543543808 bytes 2024-09-13T09:35:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1558425600 bytes 2024-09-13T09:35:55.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1619968000 bytes 2024-09-13T09:35:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1680421888 bytes 2024-09-13T09:35:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1731833856 bytes 2024-09-13T09:35:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1789332480 bytes 2024-09-13T09:35:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1875803136 bytes 2024-09-13T09:35:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1898819584 bytes 2024-09-13T09:35:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1924877312 bytes 2024-09-13T09:35:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1964378112 bytes 2024-09-13T09:35:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1976613888 bytes 2024-09-13T09:35:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1996979200 bytes 2024-09-13T09:35:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 2011652096 bytes 2024-09-13T09:35:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2012266496 bytes 2024-09-13T09:35:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2109740032 bytes 2024-09-13T09:35:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2134761472 bytes 2024-09-13T09:35:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2180470784 bytes 2024-09-13T09:35:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2218313728 bytes 2024-09-13T09:35:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2233778176 bytes 2024-09-13T09:35:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2259210240 bytes 2024-09-13T09:35:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2259823616 bytes 2024-09-13T09:35:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2260458496 bytes 2024-09-13T09:35:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2315550720 bytes 2024-09-13T09:35:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2350206976 bytes 2024-09-13T09:35:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2478663680 bytes 2024-09-13T09:35:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2571167744 bytes 2024-09-13T09:35:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2677408768 bytes 2024-09-13T09:35:58.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2679006208 bytes 2024-09-13T09:35:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2696676352 bytes 2024-09-13T09:35:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2733783040 bytes 2024-09-13T09:35:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2816509952 bytes 2024-09-13T09:35:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2837385216 bytes 2024-09-13T09:35:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2892536832 bytes 2024-09-13T09:35:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2951140352 bytes 2024-09-13T09:35:59.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 3002108928 bytes 2024-09-13T09:35:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 3064235008 bytes 2024-09-13T09:35:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 3084824576 bytes 2024-09-13T09:35:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 3151773696 bytes 2024-09-13T09:35:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 3181418496 bytes 2024-09-13T09:36:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 3276696576 bytes 2024-09-13T09:36:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 3277306880 bytes 2024-09-13T09:36:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 3335290880 bytes 2024-09-13T09:36:00.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 3370533888 bytes 2024-09-13T09:36:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3384644608 bytes 2024-09-13T09:36:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3411781632 bytes 2024-09-13T09:36:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3481440256 bytes 2024-09-13T09:36:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3531190272 bytes 2024-09-13T09:36:00.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3557369856 bytes 2024-09-13T09:36:01.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3622806528 bytes 2024-09-13T09:36:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3670918144 bytes 2024-09-13T09:36:01.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3723429888 bytes 2024-09-13T09:36:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3775810560 bytes 2024-09-13T09:36:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3867143168 bytes 2024-09-13T09:36:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3930477568 bytes 2024-09-13T09:36:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3942779904 bytes 2024-09-13T09:36:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3969501184 bytes 2024-09-13T09:36:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 4092812288 bytes 2024-09-13T09:36:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 4129716224 bytes 2024-09-13T09:36:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 4172992512 bytes 2024-09-13T09:36:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 4222389248 bytes 2024-09-13T09:36:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 4285225984 bytes 2024-09-13T09:36:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 4327784448 bytes 2024-09-13T09:36:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 4352677888 bytes 2024-09-13T09:36:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9400 IOs, 4353292288 bytes 2024-09-13T09:36:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9500 IOs, 4376015872 bytes 2024-09-13T09:36:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9600 IOs, 4386848768 bytes 2024-09-13T09:36:03.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9700 IOs, 4412213248 bytes 2024-09-13T09:36:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9800 IOs, 4483402752 bytes 2024-09-13T09:36:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9900 IOs, 4501266432 bytes 2024-09-13T09:36:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10000 IOs, 4534120448 bytes 2024-09-13T09:36:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10100 IOs, 4582728704 bytes 2024-09-13T09:36:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10200 IOs, 4597867520 bytes 2024-09-13T09:36:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10300 IOs, 4642513920 bytes 2024-09-13T09:36:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10400 IOs, 4665129984 bytes 2024-09-13T09:36:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10500 IOs, 4721265664 bytes 2024-09-13T09:36:04.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:36:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:36:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:36:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:36:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:36:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_2 2024-09-13T09:36:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2024-09-13T09:36:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:36:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:36:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:36:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_2 128M 2024-09-13T09:36:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:36:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:36:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:36:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:36:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_2 2024-09-13T09:36:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:36:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2024-09-13T09:36:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:36:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:36:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-13T09:36:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:36:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-13T09:36:05.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_2': 2024-09-13T09:36:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:36:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:36:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:36:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12843e920904 2024-09-13T09:36:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12843e920904 2024-09-13T09:36:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:36:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:36:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:36:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:36:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:36:04 2024 2024-09-13T09:36:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:36:04 2024 2024-09-13T09:36:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:36:04 2024 2024-09-13T09:36:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:36:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:36:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: c4b5aac5-000e-4a00-bd35-23bce99cad5f 2024-09-13T09:36:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:36:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:36:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-13T09:36:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:36:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:36:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=8 2024-09-13T09:36:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:36:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:36:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 36276224 bytes 2024-09-13T09:36:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 81363968 bytes 2024-09-13T09:36:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 101804032 bytes 2024-09-13T09:36:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 125227008 bytes 2024-09-13T09:36:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 159369216 bytes 2024-09-13T09:36:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 222218240 bytes 2024-09-13T09:36:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 253800448 bytes 2024-09-13T09:36:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 350507008 bytes 2024-09-13T09:36:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 399267840 bytes 2024-09-13T09:36:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 441185280 bytes 2024-09-13T09:36:06.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 448932864 bytes 2024-09-13T09:36:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 526899200 bytes 2024-09-13T09:36:06.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 563594240 bytes 2024-09-13T09:36:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 564220928 bytes 2024-09-13T09:36:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 615675904 bytes 2024-09-13T09:36:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 688330752 bytes 2024-09-13T09:36:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 770024448 bytes 2024-09-13T09:36:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 772491264 bytes 2024-09-13T09:36:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 811119616 bytes 2024-09-13T09:36:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 846418944 bytes 2024-09-13T09:36:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 919070720 bytes 2024-09-13T09:36:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 935321600 bytes 2024-09-13T09:36:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 961809408 bytes 2024-09-13T09:36:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1033767936 bytes 2024-09-13T09:36:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1069264896 bytes 2024-09-13T09:36:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1142344704 bytes 2024-09-13T09:36:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1213053952 bytes 2024-09-13T09:36:08.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1229048832 bytes 2024-09-13T09:36:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1269789696 bytes 2024-09-13T09:36:08.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1297568768 bytes 2024-09-13T09:36:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1342525440 bytes 2024-09-13T09:36:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1368846336 bytes 2024-09-13T09:36:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1414487040 bytes 2024-09-13T09:36:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1474109440 bytes 2024-09-13T09:36:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1510495232 bytes 2024-09-13T09:36:09.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1525915648 bytes 2024-09-13T09:36:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1559862272 bytes 2024-09-13T09:36:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1598920704 bytes 2024-09-13T09:36:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1669408768 bytes 2024-09-13T09:36:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1720146944 bytes 2024-09-13T09:36:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1798523904 bytes 2024-09-13T09:36:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1861404672 bytes 2024-09-13T09:36:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1892992000 bytes 2024-09-13T09:36:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1909234688 bytes 2024-09-13T09:36:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1910012928 bytes 2024-09-13T09:36:10.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2007013376 bytes 2024-09-13T09:36:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2107679744 bytes 2024-09-13T09:36:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2139062272 bytes 2024-09-13T09:36:10.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2218112000 bytes 2024-09-13T09:36:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2323270656 bytes 2024-09-13T09:36:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2383857664 bytes 2024-09-13T09:36:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2419414016 bytes 2024-09-13T09:36:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2531702784 bytes 2024-09-13T09:36:11.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2591924224 bytes 2024-09-13T09:36:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2646142976 bytes 2024-09-13T09:36:11.675 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2024-09-13T09:36:11.673+0000 7f00cac16640 -1 reset not still connected to 0x55a18f42a9c0 2024-09-13T09:36:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2646766592 bytes 2024-09-13T09:36:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2739411968 bytes 2024-09-13T09:36:11.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2795080704 bytes 2024-09-13T09:36:12.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2859988992 bytes 2024-09-13T09:36:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2878840832 bytes 2024-09-13T09:36:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2925153280 bytes 2024-09-13T09:36:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2973796352 bytes 2024-09-13T09:36:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2995501056 bytes 2024-09-13T09:36:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 3023335424 bytes 2024-09-13T09:36:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 3045668864 bytes 2024-09-13T09:36:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 3077793792 bytes 2024-09-13T09:36:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 3109223424 bytes 2024-09-13T09:36:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 3135605760 bytes 2024-09-13T09:36:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:36:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:36:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:36:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:36:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:36:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_3 2024-09-13T09:36:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2024-09-13T09:36:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:36:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:36:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:36:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_3 128M 2024-09-13T09:36:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:36:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:36:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:36:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:36:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_3 2024-09-13T09:36:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:36:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2024-09-13T09:36:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:36:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:36:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-13T09:36:14.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:36:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-13T09:36:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_3': 2024-09-13T09:36:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:36:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:36:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:36:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1290ccbc92e1 2024-09-13T09:36:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1290ccbc92e1 2024-09-13T09:36:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:36:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:36:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:36:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:36:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:36:13 2024 2024-09-13T09:36:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:36:13 2024 2024-09-13T09:36:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:36:13 2024 2024-09-13T09:36:14.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:36:14.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:36:14.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: c99ca9ae-d4dc-48db-a446-dd6058760fec 2024-09-13T09:36:14.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:36:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:36:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-13T09:36:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:36:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:36:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=9 2024-09-13T09:36:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:36:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:36:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 589824 bytes 2024-09-13T09:36:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 49223680 bytes 2024-09-13T09:36:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 71881728 bytes 2024-09-13T09:36:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 80932864 bytes 2024-09-13T09:36:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 108515328 bytes 2024-09-13T09:36:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 171362304 bytes 2024-09-13T09:36:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 216205312 bytes 2024-09-13T09:36:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 236193792 bytes 2024-09-13T09:36:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 284484608 bytes 2024-09-13T09:36:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 323204096 bytes 2024-09-13T09:36:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 360006656 bytes 2024-09-13T09:36:15.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 364376064 bytes 2024-09-13T09:36:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 410222592 bytes 2024-09-13T09:36:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 439431168 bytes 2024-09-13T09:36:15.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 461881344 bytes 2024-09-13T09:36:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 500425728 bytes 2024-09-13T09:36:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 551564288 bytes 2024-09-13T09:36:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 615309312 bytes 2024-09-13T09:36:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 615905280 bytes 2024-09-13T09:36:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 683131904 bytes 2024-09-13T09:36:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 732512256 bytes 2024-09-13T09:36:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 750666752 bytes 2024-09-13T09:36:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 773142528 bytes 2024-09-13T09:36:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 803176448 bytes 2024-09-13T09:36:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 814198784 bytes 2024-09-13T09:36:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 814809088 bytes 2024-09-13T09:36:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 849494016 bytes 2024-09-13T09:36:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 881206272 bytes 2024-09-13T09:36:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 974522368 bytes 2024-09-13T09:36:17.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1013422080 bytes 2024-09-13T09:36:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1083052032 bytes 2024-09-13T09:36:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1132828672 bytes 2024-09-13T09:36:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1152742400 bytes 2024-09-13T09:36:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1210524672 bytes 2024-09-13T09:36:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1283091456 bytes 2024-09-13T09:36:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1313825792 bytes 2024-09-13T09:36:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1387534336 bytes 2024-09-13T09:36:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1418393600 bytes 2024-09-13T09:36:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1455292416 bytes 2024-09-13T09:36:18.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1534653440 bytes 2024-09-13T09:36:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1570692096 bytes 2024-09-13T09:36:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1625454592 bytes 2024-09-13T09:36:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1695814656 bytes 2024-09-13T09:36:18.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1727661056 bytes 2024-09-13T09:36:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1744413696 bytes 2024-09-13T09:36:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1825666048 bytes 2024-09-13T09:36:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1854747648 bytes 2024-09-13T09:36:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1946887168 bytes 2024-09-13T09:36:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1984103424 bytes 2024-09-13T09:36:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2022732800 bytes 2024-09-13T09:36:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2062185472 bytes 2024-09-13T09:36:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2186685440 bytes 2024-09-13T09:36:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2212824064 bytes 2024-09-13T09:36:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2263582720 bytes 2024-09-13T09:36:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2271223808 bytes 2024-09-13T09:36:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2306266112 bytes 2024-09-13T09:36:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2334265344 bytes 2024-09-13T09:36:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2366715904 bytes 2024-09-13T09:36:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2407307264 bytes 2024-09-13T09:36:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2454327296 bytes 2024-09-13T09:36:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2475252736 bytes 2024-09-13T09:36:21.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2501277696 bytes 2024-09-13T09:36:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2539731968 bytes 2024-09-13T09:36:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2593352704 bytes 2024-09-13T09:36:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2632366080 bytes 2024-09-13T09:36:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2662130688 bytes 2024-09-13T09:36:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2719364096 bytes 2024-09-13T09:36:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2779839488 bytes 2024-09-13T09:36:21.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2807499776 bytes 2024-09-13T09:36:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2831660032 bytes 2024-09-13T09:36:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2857327616 bytes 2024-09-13T09:36:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2891233280 bytes 2024-09-13T09:36:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2945867776 bytes 2024-09-13T09:36:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 2998078464 bytes 2024-09-13T09:36:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3071362048 bytes 2024-09-13T09:36:22.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3108968448 bytes 2024-09-13T09:36:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3127861248 bytes 2024-09-13T09:36:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3147120640 bytes 2024-09-13T09:36:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3203131392 bytes 2024-09-13T09:36:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3226565632 bytes 2024-09-13T09:36:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:36:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:36:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:36:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:36:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:36:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_4 2024-09-13T09:36:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2024-09-13T09:36:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:36:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:36:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:36:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_4 128M 2024-09-13T09:36:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:36:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:36:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:36:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:36:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_4 2024-09-13T09:36:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:36:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2024-09-13T09:36:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:36:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:36:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-13T09:36:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:36:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-13T09:36:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_4': 2024-09-13T09:36:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:36:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:36:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:36:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 129cdaacb2b4 2024-09-13T09:36:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.129cdaacb2b4 2024-09-13T09:36:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:36:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:36:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:36:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:36:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:36:23 2024 2024-09-13T09:36:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:36:23 2024 2024-09-13T09:36:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:36:23 2024 2024-09-13T09:36:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:36:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:36:24.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 46dd7eaa-6bb3-4d95-bf94-ddb77bae7a9e 2024-09-13T09:36:24.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:36:24.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:36:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-13T09:36:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:36:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:36:24.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2024-09-13T09:36:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:36:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:36:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 41080832 bytes 2024-09-13T09:36:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 92592128 bytes 2024-09-13T09:36:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 105201664 bytes 2024-09-13T09:36:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 145483776 bytes 2024-09-13T09:36:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 237527040 bytes 2024-09-13T09:36:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 272549888 bytes 2024-09-13T09:36:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 347275264 bytes 2024-09-13T09:36:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 366220288 bytes 2024-09-13T09:36:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 381971456 bytes 2024-09-13T09:36:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 396059648 bytes 2024-09-13T09:36:25.391 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.37526264035371504) > weight (0.25) 2024-09-13T09:36:25.391 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7922684543086997) > weight (0.25) 2024-09-13T09:36:25.391 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4520192928833827) > weight (0.25) 2024-09-13T09:36:25.391 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4238244227125785) > weight (0.25) 2024-09-13T09:36:25.392 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2024-09-13T09:36:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 443988992 bytes 2024-09-13T09:36:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 493382656 bytes 2024-09-13T09:36:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 493983744 bytes 2024-09-13T09:36:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 559745024 bytes 2024-09-13T09:36:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 583606272 bytes 2024-09-13T09:36:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 616336384 bytes 2024-09-13T09:36:26.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 668609536 bytes 2024-09-13T09:36:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 740759552 bytes 2024-09-13T09:36:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 797072384 bytes 2024-09-13T09:36:26.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 842569728 bytes 2024-09-13T09:36:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 899158016 bytes 2024-09-13T09:36:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 942519296 bytes 2024-09-13T09:36:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 988471296 bytes 2024-09-13T09:36:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1086409728 bytes 2024-09-13T09:36:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1107291136 bytes 2024-09-13T09:36:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1228646400 bytes 2024-09-13T09:36:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1324502016 bytes 2024-09-13T09:36:27.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1349460992 bytes 2024-09-13T09:36:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1382616064 bytes 2024-09-13T09:36:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1468186624 bytes 2024-09-13T09:36:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1580178432 bytes 2024-09-13T09:36:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1592181760 bytes 2024-09-13T09:36:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1610672128 bytes 2024-09-13T09:36:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1630011392 bytes 2024-09-13T09:36:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1712434176 bytes 2024-09-13T09:36:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1730543616 bytes 2024-09-13T09:36:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1842083840 bytes 2024-09-13T09:36:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1848151040 bytes 2024-09-13T09:36:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1878427648 bytes 2024-09-13T09:36:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1899148288 bytes 2024-09-13T09:36:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1937466368 bytes 2024-09-13T09:36:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1973990400 bytes 2024-09-13T09:36:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1974620160 bytes 2024-09-13T09:36:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2015655936 bytes 2024-09-13T09:36:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2041816064 bytes 2024-09-13T09:36:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2056521728 bytes 2024-09-13T09:36:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2087785472 bytes 2024-09-13T09:36:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2123812864 bytes 2024-09-13T09:36:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2182784000 bytes 2024-09-13T09:36:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2218094592 bytes 2024-09-13T09:36:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2273826816 bytes 2024-09-13T09:36:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2318797824 bytes 2024-09-13T09:36:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2337547264 bytes 2024-09-13T09:36:30.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2351523840 bytes 2024-09-13T09:36:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2424050688 bytes 2024-09-13T09:36:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2475702272 bytes 2024-09-13T09:36:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2486370304 bytes 2024-09-13T09:36:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2531564544 bytes 2024-09-13T09:36:31.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2560903168 bytes 2024-09-13T09:36:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2580849664 bytes 2024-09-13T09:36:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2604534784 bytes 2024-09-13T09:36:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2656543744 bytes 2024-09-13T09:36:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2678250496 bytes 2024-09-13T09:36:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2774924288 bytes 2024-09-13T09:36:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2841038848 bytes 2024-09-13T09:36:31.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2907035648 bytes 2024-09-13T09:36:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2910091264 bytes 2024-09-13T09:36:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2990531584 bytes 2024-09-13T09:36:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 3100077056 bytes 2024-09-13T09:36:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 3132020736 bytes 2024-09-13T09:36:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 3132609536 bytes 2024-09-13T09:36:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 3154106368 bytes 2024-09-13T09:36:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 3169672192 bytes 2024-09-13T09:36:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3196679168 bytes 2024-09-13T09:36:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3293988864 bytes 2024-09-13T09:36:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3318793216 bytes 2024-09-13T09:36:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3337125888 bytes 2024-09-13T09:36:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3373523968 bytes 2024-09-13T09:36:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3420576768 bytes 2024-09-13T09:36:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3448039424 bytes 2024-09-13T09:36:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3475393536 bytes 2024-09-13T09:36:33.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3545355264 bytes 2024-09-13T09:36:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3552855040 bytes 2024-09-13T09:36:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3568690176 bytes 2024-09-13T09:36:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3615434752 bytes 2024-09-13T09:36:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3674318848 bytes 2024-09-13T09:36:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3688214528 bytes 2024-09-13T09:36:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3745323008 bytes 2024-09-13T09:36:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3800034304 bytes 2024-09-13T09:36:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 3855243264 bytes 2024-09-13T09:36:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 3877510144 bytes 2024-09-13T09:36:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 3919026176 bytes 2024-09-13T09:36:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 3922696192 bytes 2024-09-13T09:36:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9400 IOs, 3957342208 bytes 2024-09-13T09:36:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9500 IOs, 4068310016 bytes 2024-09-13T09:36:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9600 IOs, 4211751936 bytes 2024-09-13T09:36:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9700 IOs, 4298589184 bytes 2024-09-13T09:36:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9800 IOs, 4338578432 bytes 2024-09-13T09:36:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9900 IOs, 4358564864 bytes 2024-09-13T09:36:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:36:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:36:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:36:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:36:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:36:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_5 2024-09-13T09:36:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2024-09-13T09:36:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:36:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:36:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:36:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_5 128M 2024-09-13T09:36:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:36:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:36:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:36:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:36:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_5 2024-09-13T09:36:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:36:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2024-09-13T09:36:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:36:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:36:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-13T09:36:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:36:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-13T09:36:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_5': 2024-09-13T09:36:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:36:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:36:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:36:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12a8994fb0ec 2024-09-13T09:36:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12a8994fb0ec 2024-09-13T09:36:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:36:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:36:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:36:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:36:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:36:36 2024 2024-09-13T09:36:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:36:36 2024 2024-09-13T09:36:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:36:36 2024 2024-09-13T09:36:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:36:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:36:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 225d4f13-dab0-4960-904b-d2c54fac25f4 2024-09-13T09:36:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:36:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:36:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-13T09:36:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:36:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:36:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=8 2024-09-13T09:36:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:36:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:36:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 3975168 bytes 2024-09-13T09:36:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 77286400 bytes 2024-09-13T09:36:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 190162944 bytes 2024-09-13T09:36:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 220103680 bytes 2024-09-13T09:36:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 282317824 bytes 2024-09-13T09:36:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 301111296 bytes 2024-09-13T09:36:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 376579072 bytes 2024-09-13T09:36:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 397289472 bytes 2024-09-13T09:36:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 418857984 bytes 2024-09-13T09:36:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 463365120 bytes 2024-09-13T09:36:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 477130752 bytes 2024-09-13T09:36:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 515990528 bytes 2024-09-13T09:36:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 544926720 bytes 2024-09-13T09:36:38.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 579182592 bytes 2024-09-13T09:36:38.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 680540160 bytes 2024-09-13T09:36:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 794032128 bytes 2024-09-13T09:36:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 886301696 bytes 2024-09-13T09:36:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 942826496 bytes 2024-09-13T09:36:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 984961024 bytes 2024-09-13T09:36:39.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 1011830784 bytes 2024-09-13T09:36:39.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1025234944 bytes 2024-09-13T09:36:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1058810880 bytes 2024-09-13T09:36:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1115750400 bytes 2024-09-13T09:36:39.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1138537472 bytes 2024-09-13T09:36:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1181731840 bytes 2024-09-13T09:36:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1235941376 bytes 2024-09-13T09:36:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1276886016 bytes 2024-09-13T09:36:40.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1342556160 bytes 2024-09-13T09:36:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1359676416 bytes 2024-09-13T09:36:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1437180928 bytes 2024-09-13T09:36:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1517633536 bytes 2024-09-13T09:36:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1518229504 bytes 2024-09-13T09:36:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1571492864 bytes 2024-09-13T09:36:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1626220544 bytes 2024-09-13T09:36:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1740187648 bytes 2024-09-13T09:36:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1820053504 bytes 2024-09-13T09:36:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1862393856 bytes 2024-09-13T09:36:41.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1936544768 bytes 2024-09-13T09:36:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1972398080 bytes 2024-09-13T09:36:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 2007060480 bytes 2024-09-13T09:36:41.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 2039322624 bytes 2024-09-13T09:36:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 2080196608 bytes 2024-09-13T09:36:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 2102618112 bytes 2024-09-13T09:36:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2151464960 bytes 2024-09-13T09:36:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2158116864 bytes 2024-09-13T09:36:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2247849984 bytes 2024-09-13T09:36:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2275297280 bytes 2024-09-13T09:36:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2275910656 bytes 2024-09-13T09:36:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2276507648 bytes 2024-09-13T09:36:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2321993728 bytes 2024-09-13T09:36:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2402565120 bytes 2024-09-13T09:36:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2449106944 bytes 2024-09-13T09:36:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2482535424 bytes 2024-09-13T09:36:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2534317056 bytes 2024-09-13T09:36:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2582397952 bytes 2024-09-13T09:36:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2612917248 bytes 2024-09-13T09:36:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2624965632 bytes 2024-09-13T09:36:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2683824128 bytes 2024-09-13T09:36:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2706759680 bytes 2024-09-13T09:36:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2775572480 bytes 2024-09-13T09:36:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2831484928 bytes 2024-09-13T09:36:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2846011392 bytes 2024-09-13T09:36:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2905969664 bytes 2024-09-13T09:36:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2968992768 bytes 2024-09-13T09:36:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2973661184 bytes 2024-09-13T09:36:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:36:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:36:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:36:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:36:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:36:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_6 2024-09-13T09:36:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2024-09-13T09:36:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:36:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:36:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:36:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_6 128M 2024-09-13T09:36:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:36:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:36:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:36:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:36:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_6 2024-09-13T09:36:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:36:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2024-09-13T09:36:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:36:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:36:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-13T09:36:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:36:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-13T09:36:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_6': 2024-09-13T09:36:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:36:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:36:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:36:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12b4c73eb926 2024-09-13T09:36:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12b4c73eb926 2024-09-13T09:36:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:36:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:36:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:36:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:36:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:36:44 2024 2024-09-13T09:36:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:36:44 2024 2024-09-13T09:36:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:36:44 2024 2024-09-13T09:36:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:36:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:36:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 140af9b2-8074-4c94-96e5-2095bff8e54c 2024-09-13T09:36:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:36:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:36:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-13T09:36:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:36:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:36:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=11 2024-09-13T09:36:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:36:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:36:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 24358912 bytes 2024-09-13T09:36:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 86672384 bytes 2024-09-13T09:36:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 161033216 bytes 2024-09-13T09:36:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 185697280 bytes 2024-09-13T09:36:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 281749504 bytes 2024-09-13T09:36:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 332643328 bytes 2024-09-13T09:36:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 385851392 bytes 2024-09-13T09:36:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 386465792 bytes 2024-09-13T09:36:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 392649728 bytes 2024-09-13T09:36:46.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 428443648 bytes 2024-09-13T09:36:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 482076672 bytes 2024-09-13T09:36:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 498500608 bytes 2024-09-13T09:36:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 548911104 bytes 2024-09-13T09:36:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 559039488 bytes 2024-09-13T09:36:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 604378112 bytes 2024-09-13T09:36:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 672846848 bytes 2024-09-13T09:36:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 705664000 bytes 2024-09-13T09:36:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 770577408 bytes 2024-09-13T09:36:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 794546176 bytes 2024-09-13T09:36:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 795645952 bytes 2024-09-13T09:36:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 827665408 bytes 2024-09-13T09:36:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 847139840 bytes 2024-09-13T09:36:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 923354112 bytes 2024-09-13T09:36:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1016505344 bytes 2024-09-13T09:36:48.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1065113600 bytes 2024-09-13T09:36:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1087003648 bytes 2024-09-13T09:36:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1146124288 bytes 2024-09-13T09:36:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1256730624 bytes 2024-09-13T09:36:49.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1326518272 bytes 2024-09-13T09:36:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1352220672 bytes 2024-09-13T09:36:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1359801344 bytes 2024-09-13T09:36:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1450534912 bytes 2024-09-13T09:36:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1492963328 bytes 2024-09-13T09:36:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1572569088 bytes 2024-09-13T09:36:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1604248576 bytes 2024-09-13T09:36:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1655037952 bytes 2024-09-13T09:36:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1695391744 bytes 2024-09-13T09:36:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1769521152 bytes 2024-09-13T09:36:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1792549888 bytes 2024-09-13T09:36:50.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1809296384 bytes 2024-09-13T09:36:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1879911424 bytes 2024-09-13T09:36:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1933381632 bytes 2024-09-13T09:36:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1962417152 bytes 2024-09-13T09:36:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2049774592 bytes 2024-09-13T09:36:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2071934976 bytes 2024-09-13T09:36:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2072546304 bytes 2024-09-13T09:36:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2130950144 bytes 2024-09-13T09:36:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2163222528 bytes 2024-09-13T09:36:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2225187840 bytes 2024-09-13T09:36:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2275747840 bytes 2024-09-13T09:36:51.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2284954624 bytes 2024-09-13T09:36:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2285564928 bytes 2024-09-13T09:36:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2328667136 bytes 2024-09-13T09:36:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2382516224 bytes 2024-09-13T09:36:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2405633024 bytes 2024-09-13T09:36:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2431090688 bytes 2024-09-13T09:36:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2470369280 bytes 2024-09-13T09:36:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2504952832 bytes 2024-09-13T09:36:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2555399168 bytes 2024-09-13T09:36:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2648758272 bytes 2024-09-13T09:36:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2711566336 bytes 2024-09-13T09:36:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2730010624 bytes 2024-09-13T09:36:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2732598272 bytes 2024-09-13T09:36:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2759586816 bytes 2024-09-13T09:36:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2805645312 bytes 2024-09-13T09:36:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2839970816 bytes 2024-09-13T09:36:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2877225984 bytes 2024-09-13T09:36:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2962360320 bytes 2024-09-13T09:36:53.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 3044954112 bytes 2024-09-13T09:36:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 3131085824 bytes 2024-09-13T09:36:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 3155739648 bytes 2024-09-13T09:36:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 3206275072 bytes 2024-09-13T09:36:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 3221993472 bytes 2024-09-13T09:36:54.494 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6016558645636433) > weight (0.25) 2024-09-13T09:36:54.494 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6239067665772736) > weight (0.25) 2024-09-13T09:36:54.494 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-13T09:36:54.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-13T09:36:54.495 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-13T09:36:54.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T09:36:54.494+0000 7f9404758640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 41806) UID: 1000 2024-09-13T09:36:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3241359360 bytes 2024-09-13T09:36:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3293267968 bytes 2024-09-13T09:36:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3343237120 bytes 2024-09-13T09:36:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3374773248 bytes 2024-09-13T09:36:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3445420032 bytes 2024-09-13T09:36:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3469929472 bytes 2024-09-13T09:36:55.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3525659648 bytes 2024-09-13T09:36:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3557634048 bytes 2024-09-13T09:36:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3596475392 bytes 2024-09-13T09:36:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3631439872 bytes 2024-09-13T09:36:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3633710080 bytes 2024-09-13T09:36:55.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3714518016 bytes 2024-09-13T09:36:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3753007104 bytes 2024-09-13T09:36:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3762791424 bytes 2024-09-13T09:36:56.121 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3788243968 bytes 2024-09-13T09:36:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3881974784 bytes 2024-09-13T09:36:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 3915369472 bytes 2024-09-13T09:36:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 3966595072 bytes 2024-09-13T09:36:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:36:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:36:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:36:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:36:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:36:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_7 2024-09-13T09:36:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2024-09-13T09:36:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:36:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:36:56.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:36:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_7 128M 2024-09-13T09:36:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:36:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:36:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:36:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:36:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:36:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_7 2024-09-13T09:36:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:36:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2024-09-13T09:36:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:36:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:36:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-13T09:36:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:36:57.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-13T09:36:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_7': 2024-09-13T09:36:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:36:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:36:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:36:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12c0322cf682 2024-09-13T09:36:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12c0322cf682 2024-09-13T09:36:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:36:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:36:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:36:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:36:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:36:56 2024 2024-09-13T09:36:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:36:56 2024 2024-09-13T09:36:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:36:56 2024 2024-09-13T09:36:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:36:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:36:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 05fdfd86-7dcf-47af-bad1-344a4f52a3bd 2024-09-13T09:36:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:36:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:36:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-13T09:36:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:36:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:36:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:36:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:36:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2024-09-13T09:36:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:36:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:36:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 79947776 bytes 2024-09-13T09:36:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 86945792 bytes 2024-09-13T09:36:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 115224576 bytes 2024-09-13T09:36:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 137879552 bytes 2024-09-13T09:36:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 168441856 bytes 2024-09-13T09:36:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 233717760 bytes 2024-09-13T09:36:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 262696960 bytes 2024-09-13T09:36:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 300219392 bytes 2024-09-13T09:36:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 352957440 bytes 2024-09-13T09:36:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 392296448 bytes 2024-09-13T09:36:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 409332736 bytes 2024-09-13T09:36:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 441556992 bytes 2024-09-13T09:36:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 482653184 bytes 2024-09-13T09:36:59.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 548194304 bytes 2024-09-13T09:36:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 583377920 bytes 2024-09-13T09:36:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 612930560 bytes 2024-09-13T09:36:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 669158400 bytes 2024-09-13T09:36:59.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 669775872 bytes 2024-09-13T09:36:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 777868288 bytes 2024-09-13T09:36:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 824665088 bytes 2024-09-13T09:36:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 858126336 bytes 2024-09-13T09:37:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 889500672 bytes 2024-09-13T09:37:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 926400512 bytes 2024-09-13T09:37:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 927019008 bytes 2024-09-13T09:37:00.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 959419392 bytes 2024-09-13T09:37:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 977164288 bytes 2024-09-13T09:37:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1017096192 bytes 2024-09-13T09:37:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1036692480 bytes 2024-09-13T09:37:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1052665856 bytes 2024-09-13T09:37:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1097357312 bytes 2024-09-13T09:37:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1144368128 bytes 2024-09-13T09:37:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1189762048 bytes 2024-09-13T09:37:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1225858048 bytes 2024-09-13T09:37:01.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1264242688 bytes 2024-09-13T09:37:01.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1333135360 bytes 2024-09-13T09:37:01.495 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-13T09:37:01.496 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:37:01.496 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-13T09:37:01.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-13T09:37:01.497 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-13T09:37:01.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-13T09:37:01.500 INFO:tasks.rbd_mirror_thrash:waiting for 79 secs before thrashing 2024-09-13T09:37:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1366288384 bytes 2024-09-13T09:37:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1366920192 bytes 2024-09-13T09:37:01.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1430348800 bytes 2024-09-13T09:37:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1450992640 bytes 2024-09-13T09:37:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1496287232 bytes 2024-09-13T09:37:02.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1507085312 bytes 2024-09-13T09:37:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1534227456 bytes 2024-09-13T09:37:02.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1607079936 bytes 2024-09-13T09:37:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1645514752 bytes 2024-09-13T09:37:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1688943616 bytes 2024-09-13T09:37:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1782391808 bytes 2024-09-13T09:37:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1831358464 bytes 2024-09-13T09:37:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1850985472 bytes 2024-09-13T09:37:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1854599168 bytes 2024-09-13T09:37:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1875478528 bytes 2024-09-13T09:37:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 1923424256 bytes 2024-09-13T09:37:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2003739648 bytes 2024-09-13T09:37:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2034862080 bytes 2024-09-13T09:37:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2044951552 bytes 2024-09-13T09:37:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2078798848 bytes 2024-09-13T09:37:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2108866560 bytes 2024-09-13T09:37:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2129611776 bytes 2024-09-13T09:37:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2167504896 bytes 2024-09-13T09:37:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2226999296 bytes 2024-09-13T09:37:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2304274432 bytes 2024-09-13T09:37:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2385454080 bytes 2024-09-13T09:37:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2430326784 bytes 2024-09-13T09:37:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2498359296 bytes 2024-09-13T09:37:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2498987008 bytes 2024-09-13T09:37:04.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2528824320 bytes 2024-09-13T09:37:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2553590784 bytes 2024-09-13T09:37:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2577793024 bytes 2024-09-13T09:37:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2598533120 bytes 2024-09-13T09:37:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2617840640 bytes 2024-09-13T09:37:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2674022400 bytes 2024-09-13T09:37:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2748169216 bytes 2024-09-13T09:37:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2817411072 bytes 2024-09-13T09:37:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2868808704 bytes 2024-09-13T09:37:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 2904697856 bytes 2024-09-13T09:37:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 2970743808 bytes 2024-09-13T09:37:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 2993517568 bytes 2024-09-13T09:37:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3008749568 bytes 2024-09-13T09:37:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3042569216 bytes 2024-09-13T09:37:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3181971456 bytes 2024-09-13T09:37:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3231997952 bytes 2024-09-13T09:37:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3309418496 bytes 2024-09-13T09:37:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3346920448 bytes 2024-09-13T09:37:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3435044864 bytes 2024-09-13T09:37:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3481585664 bytes 2024-09-13T09:37:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3482184704 bytes 2024-09-13T09:37:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3529113600 bytes 2024-09-13T09:37:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3568973824 bytes 2024-09-13T09:37:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3587214336 bytes 2024-09-13T09:37:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3626826752 bytes 2024-09-13T09:37:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 3700412416 bytes 2024-09-13T09:37:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 3784838144 bytes 2024-09-13T09:37:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 3795488768 bytes 2024-09-13T09:37:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 3796092928 bytes 2024-09-13T09:37:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9400 IOs, 3864261632 bytes 2024-09-13T09:37:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9500 IOs, 3935468544 bytes 2024-09-13T09:37:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9600 IOs, 3950630912 bytes 2024-09-13T09:37:08.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9700 IOs, 4025079808 bytes 2024-09-13T09:37:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9800 IOs, 4031667200 bytes 2024-09-13T09:37:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9900 IOs, 4054538240 bytes 2024-09-13T09:37:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10000 IOs, 4086718464 bytes 2024-09-13T09:37:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10100 IOs, 4106012672 bytes 2024-09-13T09:37:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10200 IOs, 4136751104 bytes 2024-09-13T09:37:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10300 IOs, 4200476672 bytes 2024-09-13T09:37:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:37:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:37:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:37:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:37:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:37:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_8 2024-09-13T09:37:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2024-09-13T09:37:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:37:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:37:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:37:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:37:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:37:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_8 128M 2024-09-13T09:37:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:37:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:37:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:37:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:37:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:37:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:37:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_8 2024-09-13T09:37:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:37:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2024-09-13T09:37:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:37:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:37:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:37:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:37:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-13T09:37:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:37:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-13T09:37:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_8': 2024-09-13T09:37:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:37:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:37:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:37:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12d1f03b857c 2024-09-13T09:37:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12d1f03b857c 2024-09-13T09:37:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:37:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:37:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:37:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:37:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:37:09 2024 2024-09-13T09:37:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:37:09 2024 2024-09-13T09:37:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:37:09 2024 2024-09-13T09:37:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:37:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:37:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 8a9c351e-2826-4d1c-9d73-5faec463921f 2024-09-13T09:37:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:37:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:37:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-13T09:37:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:37:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:37:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:37:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:37:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=9 2024-09-13T09:37:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:37:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:37:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 15169536 bytes 2024-09-13T09:37:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 30467072 bytes 2024-09-13T09:37:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 77903872 bytes 2024-09-13T09:37:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 103907328 bytes 2024-09-13T09:37:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 199462912 bytes 2024-09-13T09:37:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 236468224 bytes 2024-09-13T09:37:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 262016000 bytes 2024-09-13T09:37:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 342208512 bytes 2024-09-13T09:37:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 404133888 bytes 2024-09-13T09:37:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 424587264 bytes 2024-09-13T09:37:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 493347840 bytes 2024-09-13T09:37:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 524596224 bytes 2024-09-13T09:37:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 554583040 bytes 2024-09-13T09:37:12.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 599563264 bytes 2024-09-13T09:37:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 608740352 bytes 2024-09-13T09:37:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 610989056 bytes 2024-09-13T09:37:12.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 650969088 bytes 2024-09-13T09:37:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 678809600 bytes 2024-09-13T09:37:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 724219904 bytes 2024-09-13T09:37:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 769796096 bytes 2024-09-13T09:37:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 803656704 bytes 2024-09-13T09:37:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 846780416 bytes 2024-09-13T09:37:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 862667776 bytes 2024-09-13T09:37:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 941155328 bytes 2024-09-13T09:37:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1009539072 bytes 2024-09-13T09:37:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1030774784 bytes 2024-09-13T09:37:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1080658944 bytes 2024-09-13T09:37:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1147243520 bytes 2024-09-13T09:37:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1162466304 bytes 2024-09-13T09:37:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1182928896 bytes 2024-09-13T09:37:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1239706624 bytes 2024-09-13T09:37:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1284292608 bytes 2024-09-13T09:37:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1355740160 bytes 2024-09-13T09:37:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1366199296 bytes 2024-09-13T09:37:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1445306368 bytes 2024-09-13T09:37:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1547443200 bytes 2024-09-13T09:37:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1587145728 bytes 2024-09-13T09:37:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1613220864 bytes 2024-09-13T09:37:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1644320768 bytes 2024-09-13T09:37:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1687109632 bytes 2024-09-13T09:37:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1717081088 bytes 2024-09-13T09:37:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1756123136 bytes 2024-09-13T09:37:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1809795072 bytes 2024-09-13T09:37:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1839821824 bytes 2024-09-13T09:37:16.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1895864320 bytes 2024-09-13T09:37:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1915441152 bytes 2024-09-13T09:37:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1938703360 bytes 2024-09-13T09:37:16.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2019735552 bytes 2024-09-13T09:37:16.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2030156800 bytes 2024-09-13T09:37:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2068983808 bytes 2024-09-13T09:37:16.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2112231424 bytes 2024-09-13T09:37:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2119973888 bytes 2024-09-13T09:37:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2120598528 bytes 2024-09-13T09:37:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2136276992 bytes 2024-09-13T09:37:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2155193344 bytes 2024-09-13T09:37:17.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2213766144 bytes 2024-09-13T09:37:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2279223296 bytes 2024-09-13T09:37:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2291910656 bytes 2024-09-13T09:37:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2429389824 bytes 2024-09-13T09:37:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2452436992 bytes 2024-09-13T09:37:18.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2527413248 bytes 2024-09-13T09:37:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2587575296 bytes 2024-09-13T09:37:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2616402944 bytes 2024-09-13T09:37:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2667360256 bytes 2024-09-13T09:37:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2721352704 bytes 2024-09-13T09:37:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2744549376 bytes 2024-09-13T09:37:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2787080192 bytes 2024-09-13T09:37:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2828123136 bytes 2024-09-13T09:37:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2829326336 bytes 2024-09-13T09:37:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2880470016 bytes 2024-09-13T09:37:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2894916608 bytes 2024-09-13T09:37:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2946234368 bytes 2024-09-13T09:37:19.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2978270208 bytes 2024-09-13T09:37:19.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3018510336 bytes 2024-09-13T09:37:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:37:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:37:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:37:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:37:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:37:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_9 2024-09-13T09:37:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2024-09-13T09:37:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:37:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:37:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:37:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:37:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:37:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_9 128M 2024-09-13T09:37:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:37:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:37:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:37:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:37:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:37:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:37:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_9 2024-09-13T09:37:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:37:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2024-09-13T09:37:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:37:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:37:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:37:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:37:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-13T09:37:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:37:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-13T09:37:20.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_9': 2024-09-13T09:37:20.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:37:20.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:37:20.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:37:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12db6c48d46a 2024-09-13T09:37:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12db6c48d46a 2024-09-13T09:37:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:37:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:37:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:37:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:37:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:37:19 2024 2024-09-13T09:37:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:37:19 2024 2024-09-13T09:37:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:37:19 2024 2024-09-13T09:37:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:37:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:37:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: ce3d15a2-54a3-4901-a2c5-f75b935cc27e 2024-09-13T09:37:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:37:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:37:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-13T09:37:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:37:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:37:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:37:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:37:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=7 2024-09-13T09:37:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:37:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:37:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 55447552 bytes 2024-09-13T09:37:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 114826240 bytes 2024-09-13T09:37:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 134418432 bytes 2024-09-13T09:37:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 218822656 bytes 2024-09-13T09:37:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 297921536 bytes 2024-09-13T09:37:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 298498048 bytes 2024-09-13T09:37:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 324112384 bytes 2024-09-13T09:37:21.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 369139712 bytes 2024-09-13T09:37:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 406032384 bytes 2024-09-13T09:37:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 537426944 bytes 2024-09-13T09:37:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 597886976 bytes 2024-09-13T09:37:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 637626368 bytes 2024-09-13T09:37:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 675062784 bytes 2024-09-13T09:37:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 689697792 bytes 2024-09-13T09:37:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 690302976 bytes 2024-09-13T09:37:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 692114432 bytes 2024-09-13T09:37:22.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 730691584 bytes 2024-09-13T09:37:22.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 755074048 bytes 2024-09-13T09:37:22.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 838692864 bytes 2024-09-13T09:37:23.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 857559040 bytes 2024-09-13T09:37:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 901356544 bytes 2024-09-13T09:37:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 978620416 bytes 2024-09-13T09:37:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1039835136 bytes 2024-09-13T09:37:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1040442368 bytes 2024-09-13T09:37:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1059332096 bytes 2024-09-13T09:37:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1112003584 bytes 2024-09-13T09:37:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1174523904 bytes 2024-09-13T09:37:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1217893376 bytes 2024-09-13T09:37:24.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1218519040 bytes 2024-09-13T09:37:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1277891584 bytes 2024-09-13T09:37:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1303359488 bytes 2024-09-13T09:37:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1303979008 bytes 2024-09-13T09:37:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1304564736 bytes 2024-09-13T09:37:24.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1313286144 bytes 2024-09-13T09:37:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1346945024 bytes 2024-09-13T09:37:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1361375232 bytes 2024-09-13T09:37:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1419544576 bytes 2024-09-13T09:37:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1458411520 bytes 2024-09-13T09:37:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1477373952 bytes 2024-09-13T09:37:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1528281088 bytes 2024-09-13T09:37:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1632071680 bytes 2024-09-13T09:37:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1688596480 bytes 2024-09-13T09:37:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1735243776 bytes 2024-09-13T09:37:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1773379584 bytes 2024-09-13T09:37:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1816197120 bytes 2024-09-13T09:37:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1866014720 bytes 2024-09-13T09:37:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1950012416 bytes 2024-09-13T09:37:26.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1986174976 bytes 2024-09-13T09:37:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1993920512 bytes 2024-09-13T09:37:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2032559104 bytes 2024-09-13T09:37:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2058484736 bytes 2024-09-13T09:37:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2125732864 bytes 2024-09-13T09:37:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2239831040 bytes 2024-09-13T09:37:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2268779520 bytes 2024-09-13T09:37:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2297053184 bytes 2024-09-13T09:37:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2352187392 bytes 2024-09-13T09:37:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2376416256 bytes 2024-09-13T09:37:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2377003008 bytes 2024-09-13T09:37:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2383270912 bytes 2024-09-13T09:37:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:37:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:37:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:37:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:37:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:37:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_10 2024-09-13T09:37:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2024-09-13T09:37:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:37:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:37:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:37:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:37:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:37:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_10 128M 2024-09-13T09:37:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:37:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:37:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:37:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:37:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:37:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:37:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_10 2024-09-13T09:37:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:37:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2024-09-13T09:37:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:37:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:37:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:37:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:37:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-13T09:37:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:37:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-13T09:37:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_10': 2024-09-13T09:37:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:37:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:37:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:37:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12e45f20ce92 2024-09-13T09:37:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12e45f20ce92 2024-09-13T09:37:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:37:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:37:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:37:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:37:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:37:27 2024 2024-09-13T09:37:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:37:27 2024 2024-09-13T09:37:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:37:27 2024 2024-09-13T09:37:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:37:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:37:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: dd6c9225-99be-46ca-b40c-23bfa9e8daa3 2024-09-13T09:37:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:37:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:37:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-13T09:37:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:37:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:37:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:37:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:37:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=11 2024-09-13T09:37:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:37:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:37:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 59675648 bytes 2024-09-13T09:37:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 91764736 bytes 2024-09-13T09:37:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 141148160 bytes 2024-09-13T09:37:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 218995712 bytes 2024-09-13T09:37:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 219614208 bytes 2024-09-13T09:37:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 233568256 bytes 2024-09-13T09:37:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 293069824 bytes 2024-09-13T09:37:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 307145728 bytes 2024-09-13T09:37:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 360875008 bytes 2024-09-13T09:37:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 361477120 bytes 2024-09-13T09:37:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 415122432 bytes 2024-09-13T09:37:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 447888384 bytes 2024-09-13T09:37:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 486245376 bytes 2024-09-13T09:37:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 509012992 bytes 2024-09-13T09:37:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 523250688 bytes 2024-09-13T09:37:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 596771840 bytes 2024-09-13T09:37:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 610841600 bytes 2024-09-13T09:37:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 611432448 bytes 2024-09-13T09:37:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 626143232 bytes 2024-09-13T09:37:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 664038400 bytes 2024-09-13T09:37:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 676997120 bytes 2024-09-13T09:37:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 736891904 bytes 2024-09-13T09:37:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 757120000 bytes 2024-09-13T09:37:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 777068544 bytes 2024-09-13T09:37:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 789084160 bytes 2024-09-13T09:37:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 817631232 bytes 2024-09-13T09:37:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 860206080 bytes 2024-09-13T09:37:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 906766336 bytes 2024-09-13T09:37:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 928232448 bytes 2024-09-13T09:37:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 953949184 bytes 2024-09-13T09:37:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 958846976 bytes 2024-09-13T09:37:32.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1011309568 bytes 2024-09-13T09:37:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1056880640 bytes 2024-09-13T09:37:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1116187648 bytes 2024-09-13T09:37:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1198085120 bytes 2024-09-13T09:37:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1286236160 bytes 2024-09-13T09:37:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1341175808 bytes 2024-09-13T09:37:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1391080448 bytes 2024-09-13T09:37:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1440628736 bytes 2024-09-13T09:37:33.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1546757120 bytes 2024-09-13T09:37:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1585363968 bytes 2024-09-13T09:37:33.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1603670016 bytes 2024-09-13T09:37:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1637643264 bytes 2024-09-13T09:37:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1675947008 bytes 2024-09-13T09:37:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1721241600 bytes 2024-09-13T09:37:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1767412736 bytes 2024-09-13T09:37:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1768043520 bytes 2024-09-13T09:37:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1816704000 bytes 2024-09-13T09:37:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1867506688 bytes 2024-09-13T09:37:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1964229632 bytes 2024-09-13T09:37:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2009050112 bytes 2024-09-13T09:37:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2071164928 bytes 2024-09-13T09:37:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2112409600 bytes 2024-09-13T09:37:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2164037632 bytes 2024-09-13T09:37:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2251255808 bytes 2024-09-13T09:37:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2337830912 bytes 2024-09-13T09:37:35.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2392554496 bytes 2024-09-13T09:37:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2451280896 bytes 2024-09-13T09:37:35.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2504937472 bytes 2024-09-13T09:37:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2557285376 bytes 2024-09-13T09:37:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2642053120 bytes 2024-09-13T09:37:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2656336896 bytes 2024-09-13T09:37:36.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2684393472 bytes 2024-09-13T09:37:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2714594304 bytes 2024-09-13T09:37:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2799363072 bytes 2024-09-13T09:37:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2837618688 bytes 2024-09-13T09:37:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2893791232 bytes 2024-09-13T09:37:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2973073408 bytes 2024-09-13T09:37:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 3060430848 bytes 2024-09-13T09:37:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 3110179840 bytes 2024-09-13T09:37:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 3164489728 bytes 2024-09-13T09:37:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 3233299456 bytes 2024-09-13T09:37:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 3249462272 bytes 2024-09-13T09:37:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3309437952 bytes 2024-09-13T09:37:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3344882688 bytes 2024-09-13T09:37:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3364893696 bytes 2024-09-13T09:37:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3447419904 bytes 2024-09-13T09:37:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3460089856 bytes 2024-09-13T09:37:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3514059776 bytes 2024-09-13T09:37:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3520096256 bytes 2024-09-13T09:37:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3530335232 bytes 2024-09-13T09:37:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3530933248 bytes 2024-09-13T09:37:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3531540480 bytes 2024-09-13T09:37:38.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3569771520 bytes 2024-09-13T09:37:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3590049792 bytes 2024-09-13T09:37:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3599396864 bytes 2024-09-13T09:37:39.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3642811392 bytes 2024-09-13T09:37:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3687176192 bytes 2024-09-13T09:37:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3811025920 bytes 2024-09-13T09:37:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 3896166400 bytes 2024-09-13T09:37:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:37:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:37:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:37:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:37:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:37:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_11 2024-09-13T09:37:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2024-09-13T09:37:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:37:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:37:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:37:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:37:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:37:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_11 128M 2024-09-13T09:37:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:37:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:37:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:37:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:37:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:37:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:37:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_11 2024-09-13T09:37:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:37:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2024-09-13T09:37:39.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:37:39.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:37:39.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:37:39.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:37:39.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2024-09-13T09:37:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:37:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2024-09-13T09:37:40.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_11': 2024-09-13T09:37:40.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:37:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:37:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:37:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12ec7465b691 2024-09-13T09:37:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12ec7465b691 2024-09-13T09:37:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:37:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:37:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:37:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:37:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:37:39 2024 2024-09-13T09:37:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:37:39 2024 2024-09-13T09:37:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:37:39 2024 2024-09-13T09:37:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:37:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:37:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: e78a9a0a-006b-4a90-8dde-2667b994e2f6 2024-09-13T09:37:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:37:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:37:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_11 2024-09-13T09:37:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:37:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:37:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:37:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:37:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=9 2024-09-13T09:37:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:37:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:37:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 28462080 bytes 2024-09-13T09:37:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 66974720 bytes 2024-09-13T09:37:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 95104000 bytes 2024-09-13T09:37:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 105106432 bytes 2024-09-13T09:37:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 148034560 bytes 2024-09-13T09:37:41.393 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5291559847143991) > weight (0.25) 2024-09-13T09:37:41.393 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.305505619748212) > weight (0.25) 2024-09-13T09:37:41.393 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8794972181807404) > weight (0.25) 2024-09-13T09:37:41.393 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8433777468602183) > weight (0.25) 2024-09-13T09:37:41.394 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2024-09-13T09:37:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 164386816 bytes 2024-09-13T09:37:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 191082496 bytes 2024-09-13T09:37:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 248573952 bytes 2024-09-13T09:37:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 319420416 bytes 2024-09-13T09:37:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 403459072 bytes 2024-09-13T09:37:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 455933952 bytes 2024-09-13T09:37:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 474126336 bytes 2024-09-13T09:37:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 474752000 bytes 2024-09-13T09:37:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 488623104 bytes 2024-09-13T09:37:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 514491392 bytes 2024-09-13T09:37:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 582471680 bytes 2024-09-13T09:37:42.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 600953856 bytes 2024-09-13T09:37:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 649674752 bytes 2024-09-13T09:37:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 684846080 bytes 2024-09-13T09:37:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 706709504 bytes 2024-09-13T09:37:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 744858624 bytes 2024-09-13T09:37:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 863636480 bytes 2024-09-13T09:37:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 939217920 bytes 2024-09-13T09:37:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 996762624 bytes 2024-09-13T09:37:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1025364992 bytes 2024-09-13T09:37:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1092276224 bytes 2024-09-13T09:37:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1113400320 bytes 2024-09-13T09:37:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1138281472 bytes 2024-09-13T09:37:44.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1174544384 bytes 2024-09-13T09:37:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1224301568 bytes 2024-09-13T09:37:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1274297344 bytes 2024-09-13T09:37:44.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1408790528 bytes 2024-09-13T09:37:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1422065664 bytes 2024-09-13T09:37:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1435640832 bytes 2024-09-13T09:37:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1486234624 bytes 2024-09-13T09:37:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1518225408 bytes 2024-09-13T09:37:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1539007488 bytes 2024-09-13T09:37:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1661923328 bytes 2024-09-13T09:37:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1676626944 bytes 2024-09-13T09:37:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1677227008 bytes 2024-09-13T09:37:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1722273792 bytes 2024-09-13T09:37:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1762528256 bytes 2024-09-13T09:37:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1769176064 bytes 2024-09-13T09:37:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1822631936 bytes 2024-09-13T09:37:46.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1920942080 bytes 2024-09-13T09:37:46.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1972214784 bytes 2024-09-13T09:37:46.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2004236288 bytes 2024-09-13T09:37:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2056952832 bytes 2024-09-13T09:37:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2080660480 bytes 2024-09-13T09:37:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2183250944 bytes 2024-09-13T09:37:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2213716992 bytes 2024-09-13T09:37:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2292311040 bytes 2024-09-13T09:37:47.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2292919296 bytes 2024-09-13T09:37:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2317064192 bytes 2024-09-13T09:37:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2374665216 bytes 2024-09-13T09:37:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2375273472 bytes 2024-09-13T09:37:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2453237760 bytes 2024-09-13T09:37:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2457121792 bytes 2024-09-13T09:37:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2535078912 bytes 2024-09-13T09:37:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2605170688 bytes 2024-09-13T09:37:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2639887360 bytes 2024-09-13T09:37:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2701598720 bytes 2024-09-13T09:37:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2782132224 bytes 2024-09-13T09:37:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2836720640 bytes 2024-09-13T09:37:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2849740800 bytes 2024-09-13T09:37:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2862884864 bytes 2024-09-13T09:37:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2891173888 bytes 2024-09-13T09:37:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2985716736 bytes 2024-09-13T09:37:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 3121042432 bytes 2024-09-13T09:37:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:37:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:37:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:37:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:37:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:37:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_12 2024-09-13T09:37:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2024-09-13T09:37:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:37:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:37:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:37:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:37:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:37:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_12 128M 2024-09-13T09:37:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:37:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:37:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:37:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:49.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:37:49.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:37:49.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:37:49.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:37:49.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_12 2024-09-13T09:37:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:37:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2024-09-13T09:37:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:37:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:37:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:37:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:37:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2024-09-13T09:37:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:37:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2024-09-13T09:37:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_12': 2024-09-13T09:37:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:37:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:37:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:37:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12f5c0e901db 2024-09-13T09:37:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12f5c0e901db 2024-09-13T09:37:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:37:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:37:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:37:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:37:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:37:49 2024 2024-09-13T09:37:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:37:49 2024 2024-09-13T09:37:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:37:49 2024 2024-09-13T09:37:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:37:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:37:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: d677823f-dea5-4da8-94e6-42070cc16fc2 2024-09-13T09:37:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:37:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:37:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_12 2024-09-13T09:37:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:37:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:37:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:37:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:37:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=13 2024-09-13T09:37:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:37:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:37:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 91326464 bytes 2024-09-13T09:37:51.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 114421760 bytes 2024-09-13T09:37:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 186502144 bytes 2024-09-13T09:37:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 220224512 bytes 2024-09-13T09:37:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 220819456 bytes 2024-09-13T09:37:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 286513152 bytes 2024-09-13T09:37:51.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 342813696 bytes 2024-09-13T09:37:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 401766400 bytes 2024-09-13T09:37:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 462691328 bytes 2024-09-13T09:37:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 505053184 bytes 2024-09-13T09:37:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 567738368 bytes 2024-09-13T09:37:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 647967744 bytes 2024-09-13T09:37:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 711387136 bytes 2024-09-13T09:37:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 712020992 bytes 2024-09-13T09:37:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 805156864 bytes 2024-09-13T09:37:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 887241728 bytes 2024-09-13T09:37:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 942422016 bytes 2024-09-13T09:37:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 972160000 bytes 2024-09-13T09:37:53.264 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2024-09-13T09:37:53.262+0000 7f3610499640 -1 reset not still connected to 0x55cdebd8a0d0 2024-09-13T09:37:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 1019945984 bytes 2024-09-13T09:37:53.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 1074348032 bytes 2024-09-13T09:37:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1108196352 bytes 2024-09-13T09:37:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1161119744 bytes 2024-09-13T09:37:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1200259072 bytes 2024-09-13T09:37:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1255361536 bytes 2024-09-13T09:37:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1328595968 bytes 2024-09-13T09:37:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1384693760 bytes 2024-09-13T09:37:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1435761664 bytes 2024-09-13T09:37:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1488237568 bytes 2024-09-13T09:37:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1548786688 bytes 2024-09-13T09:37:54.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1564526592 bytes 2024-09-13T09:37:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1581715456 bytes 2024-09-13T09:37:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1655289856 bytes 2024-09-13T09:37:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1706684416 bytes 2024-09-13T09:37:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1707275264 bytes 2024-09-13T09:37:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1723394048 bytes 2024-09-13T09:37:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1741623296 bytes 2024-09-13T09:37:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1823049728 bytes 2024-09-13T09:37:55.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1882916864 bytes 2024-09-13T09:37:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1939593216 bytes 2024-09-13T09:37:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1989055488 bytes 2024-09-13T09:37:55.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 2041102336 bytes 2024-09-13T09:37:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 2145390592 bytes 2024-09-13T09:37:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 2171324416 bytes 2024-09-13T09:37:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2217888768 bytes 2024-09-13T09:37:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2358762496 bytes 2024-09-13T09:37:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2422484992 bytes 2024-09-13T09:37:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2435129344 bytes 2024-09-13T09:37:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2471351296 bytes 2024-09-13T09:37:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2529604608 bytes 2024-09-13T09:37:57.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2587067392 bytes 2024-09-13T09:37:57.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2657496064 bytes 2024-09-13T09:37:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2713745408 bytes 2024-09-13T09:37:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2763881472 bytes 2024-09-13T09:37:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2802007040 bytes 2024-09-13T09:37:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2861318144 bytes 2024-09-13T09:37:58.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2901414912 bytes 2024-09-13T09:37:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2915267584 bytes 2024-09-13T09:37:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2958635008 bytes 2024-09-13T09:37:58.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2959241216 bytes 2024-09-13T09:37:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 3014584320 bytes 2024-09-13T09:37:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 3045946368 bytes 2024-09-13T09:37:58.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 3113447424 bytes 2024-09-13T09:37:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 3149211648 bytes 2024-09-13T09:37:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 3159271424 bytes 2024-09-13T09:37:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 3195460608 bytes 2024-09-13T09:37:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 3222024192 bytes 2024-09-13T09:37:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 3274087424 bytes 2024-09-13T09:37:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 3332048896 bytes 2024-09-13T09:37:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 3338728448 bytes 2024-09-13T09:37:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 3482707968 bytes 2024-09-13T09:37:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 3508384768 bytes 2024-09-13T09:37:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 3529858048 bytes 2024-09-13T09:37:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 3543205888 bytes 2024-09-13T09:38:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3570037760 bytes 2024-09-13T09:38:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3587110912 bytes 2024-09-13T09:38:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3594210304 bytes 2024-09-13T09:38:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3625824256 bytes 2024-09-13T09:38:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3688268800 bytes 2024-09-13T09:38:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3731344384 bytes 2024-09-13T09:38:00.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3801027584 bytes 2024-09-13T09:38:00.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3842168832 bytes 2024-09-13T09:38:01.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3883882496 bytes 2024-09-13T09:38:01.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 4025409536 bytes 2024-09-13T09:38:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 4084559872 bytes 2024-09-13T09:38:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 4148037632 bytes 2024-09-13T09:38:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 4209442816 bytes 2024-09-13T09:38:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 4252432384 bytes 2024-09-13T09:38:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 4301634560 bytes 2024-09-13T09:38:02.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 4315935744 bytes 2024-09-13T09:38:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 4436198400 bytes 2024-09-13T09:38:02.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 4513912832 bytes 2024-09-13T09:38:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 4585712640 bytes 2024-09-13T09:38:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 4616538112 bytes 2024-09-13T09:38:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9400 IOs, 4686511104 bytes 2024-09-13T09:38:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9500 IOs, 4761114624 bytes 2024-09-13T09:38:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9600 IOs, 4794143744 bytes 2024-09-13T09:38:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9700 IOs, 4856519680 bytes 2024-09-13T09:38:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9800 IOs, 4860153856 bytes 2024-09-13T09:38:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9900 IOs, 4913743872 bytes 2024-09-13T09:38:03.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10000 IOs, 4958832640 bytes 2024-09-13T09:38:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10100 IOs, 4959414272 bytes 2024-09-13T09:38:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:38:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:38:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:38:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:38:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:38:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_13 2024-09-13T09:38:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2024-09-13T09:38:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:38:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:38:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:38:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_13 128M 2024-09-13T09:38:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:38:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:38:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:38:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:38:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:03.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_13 2024-09-13T09:38:03.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:38:03.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2024-09-13T09:38:03.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:03.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:03.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:38:03.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:38:03.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2024-09-13T09:38:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:38:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2024-09-13T09:38:04.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_13': 2024-09-13T09:38:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:38:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:38:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:38:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12ffd4f6c502 2024-09-13T09:38:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12ffd4f6c502 2024-09-13T09:38:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:38:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:38:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:38:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:38:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:38:03 2024 2024-09-13T09:38:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:38:03 2024 2024-09-13T09:38:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:38:03 2024 2024-09-13T09:38:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:38:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:38:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: fa86eec0-62a5-41fc-9cc3-aa9334a25549 2024-09-13T09:38:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:38:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:38:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_13 2024-09-13T09:38:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:38:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:38:04.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=6 2024-09-13T09:38:04.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:38:04.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:38:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 55029760 bytes 2024-09-13T09:38:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 108533760 bytes 2024-09-13T09:38:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 136094720 bytes 2024-09-13T09:38:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 157165568 bytes 2024-09-13T09:38:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 216048640 bytes 2024-09-13T09:38:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 261001216 bytes 2024-09-13T09:38:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 280585216 bytes 2024-09-13T09:38:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 355731456 bytes 2024-09-13T09:38:05.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 367792128 bytes 2024-09-13T09:38:05.944 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2024-09-13T09:38:05.942+0000 7ff85b706640 -1 reset not still connected to 0x5568f02b5ad0 2024-09-13T09:38:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 422715392 bytes 2024-09-13T09:38:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 489594880 bytes 2024-09-13T09:38:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 528760832 bytes 2024-09-13T09:38:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 569237504 bytes 2024-09-13T09:38:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 607006720 bytes 2024-09-13T09:38:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 656178176 bytes 2024-09-13T09:38:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 704562176 bytes 2024-09-13T09:38:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 724251648 bytes 2024-09-13T09:38:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 774151168 bytes 2024-09-13T09:38:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 812695552 bytes 2024-09-13T09:38:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 926553088 bytes 2024-09-13T09:38:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 978514944 bytes 2024-09-13T09:38:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1037600768 bytes 2024-09-13T09:38:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1051153408 bytes 2024-09-13T09:38:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1087040512 bytes 2024-09-13T09:38:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1107972096 bytes 2024-09-13T09:38:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1174145024 bytes 2024-09-13T09:38:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1206795264 bytes 2024-09-13T09:38:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1261468672 bytes 2024-09-13T09:38:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1286968320 bytes 2024-09-13T09:38:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1287579648 bytes 2024-09-13T09:38:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1377092608 bytes 2024-09-13T09:38:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1401018368 bytes 2024-09-13T09:38:08.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1454186496 bytes 2024-09-13T09:38:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1495435264 bytes 2024-09-13T09:38:09.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1525275648 bytes 2024-09-13T09:38:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1604790272 bytes 2024-09-13T09:38:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1632142336 bytes 2024-09-13T09:38:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1641558016 bytes 2024-09-13T09:38:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1687806976 bytes 2024-09-13T09:38:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1746589696 bytes 2024-09-13T09:38:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1805514752 bytes 2024-09-13T09:38:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1855560704 bytes 2024-09-13T09:38:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1884041216 bytes 2024-09-13T09:38:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1974313984 bytes 2024-09-13T09:38:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1974929408 bytes 2024-09-13T09:38:10.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2035566592 bytes 2024-09-13T09:38:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2086349824 bytes 2024-09-13T09:38:10.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:38:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:38:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:38:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:38:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:38:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_14 2024-09-13T09:38:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2024-09-13T09:38:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:38:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:38:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:38:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_14 128M 2024-09-13T09:38:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:38:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:38:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:38:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:38:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_14 2024-09-13T09:38:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:38:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2024-09-13T09:38:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:38:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:38:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2024-09-13T09:38:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:38:11.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2024-09-13T09:38:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_14': 2024-09-13T09:38:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:38:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:38:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:38:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 130979d66f47 2024-09-13T09:38:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.130979d66f47 2024-09-13T09:38:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:38:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:38:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:38:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:38:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:38:10 2024 2024-09-13T09:38:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:38:10 2024 2024-09-13T09:38:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:38:10 2024 2024-09-13T09:38:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:38:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:38:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 393ab671-e876-42d3-878c-4b6c73393fe3 2024-09-13T09:38:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:38:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:38:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_14 2024-09-13T09:38:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:38:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:38:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=9 2024-09-13T09:38:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:38:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:38:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 37496832 bytes 2024-09-13T09:38:12.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 80052224 bytes 2024-09-13T09:38:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 85426176 bytes 2024-09-13T09:38:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 157389824 bytes 2024-09-13T09:38:12.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 196245504 bytes 2024-09-13T09:38:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 226796544 bytes 2024-09-13T09:38:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 250159104 bytes 2024-09-13T09:38:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 296491008 bytes 2024-09-13T09:38:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 333669376 bytes 2024-09-13T09:38:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 378976256 bytes 2024-09-13T09:38:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 393081856 bytes 2024-09-13T09:38:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 440353792 bytes 2024-09-13T09:38:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 501842944 bytes 2024-09-13T09:38:13.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 510891008 bytes 2024-09-13T09:38:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 526248960 bytes 2024-09-13T09:38:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 585559040 bytes 2024-09-13T09:38:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 611987456 bytes 2024-09-13T09:38:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 679913472 bytes 2024-09-13T09:38:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 734173184 bytes 2024-09-13T09:38:14.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 734800896 bytes 2024-09-13T09:38:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 795010048 bytes 2024-09-13T09:38:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 884395008 bytes 2024-09-13T09:38:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 963343360 bytes 2024-09-13T09:38:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1071305728 bytes 2024-09-13T09:38:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1124783104 bytes 2024-09-13T09:38:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1192778752 bytes 2024-09-13T09:38:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1217675264 bytes 2024-09-13T09:38:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1295677440 bytes 2024-09-13T09:38:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1339636736 bytes 2024-09-13T09:38:15.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1354917888 bytes 2024-09-13T09:38:15.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1380909056 bytes 2024-09-13T09:38:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1410301952 bytes 2024-09-13T09:38:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1490472960 bytes 2024-09-13T09:38:16.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1522571264 bytes 2024-09-13T09:38:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1561404416 bytes 2024-09-13T09:38:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1580694528 bytes 2024-09-13T09:38:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1615053824 bytes 2024-09-13T09:38:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1630336000 bytes 2024-09-13T09:38:16.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1697592320 bytes 2024-09-13T09:38:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1724249088 bytes 2024-09-13T09:38:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1736356864 bytes 2024-09-13T09:38:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1768228864 bytes 2024-09-13T09:38:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1813683200 bytes 2024-09-13T09:38:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1849799680 bytes 2024-09-13T09:38:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1894584320 bytes 2024-09-13T09:38:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1906243584 bytes 2024-09-13T09:38:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1931782144 bytes 2024-09-13T09:38:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1990422528 bytes 2024-09-13T09:38:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2029776896 bytes 2024-09-13T09:38:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2065011712 bytes 2024-09-13T09:38:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2070167552 bytes 2024-09-13T09:38:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2133049344 bytes 2024-09-13T09:38:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2217627648 bytes 2024-09-13T09:38:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2308320256 bytes 2024-09-13T09:38:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2323705856 bytes 2024-09-13T09:38:18.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2363965440 bytes 2024-09-13T09:38:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2405704704 bytes 2024-09-13T09:38:19.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2427435008 bytes 2024-09-13T09:38:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2446317568 bytes 2024-09-13T09:38:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2490645504 bytes 2024-09-13T09:38:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2518092800 bytes 2024-09-13T09:38:19.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2552633344 bytes 2024-09-13T09:38:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2581642240 bytes 2024-09-13T09:38:19.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2618241024 bytes 2024-09-13T09:38:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2660200448 bytes 2024-09-13T09:38:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2720132096 bytes 2024-09-13T09:38:20.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2743424000 bytes 2024-09-13T09:38:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2779547648 bytes 2024-09-13T09:38:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2793976832 bytes 2024-09-13T09:38:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2794577920 bytes 2024-09-13T09:38:20.501 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.26128969270407243) > weight (0.25) 2024-09-13T09:38:20.501 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-13T09:38:20.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-13T09:38:20.502 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-13T09:38:20.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T09:38:20.501+0000 7fc701189640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 42389) UID: 1000 2024-09-13T09:38:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2826379264 bytes 2024-09-13T09:38:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2890036224 bytes 2024-09-13T09:38:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:38:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:38:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:38:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:38:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:38:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_15 2024-09-13T09:38:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2024-09-13T09:38:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:38:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:38:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:38:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_15 128M 2024-09-13T09:38:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:38:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:38:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:38:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:38:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_15 2024-09-13T09:38:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:38:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2024-09-13T09:38:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:38:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:38:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2024-09-13T09:38:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:38:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2024-09-13T09:38:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_15': 2024-09-13T09:38:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:38:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:38:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:38:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1311aef1f66b 2024-09-13T09:38:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1311aef1f66b 2024-09-13T09:38:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:38:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:38:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:38:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:38:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:38:20 2024 2024-09-13T09:38:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:38:20 2024 2024-09-13T09:38:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:38:20 2024 2024-09-13T09:38:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:38:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:38:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 22b97631-339a-40e4-93c4-03fc33081710 2024-09-13T09:38:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:38:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:38:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_15 2024-09-13T09:38:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:38:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:38:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=8 2024-09-13T09:38:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:38:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:38:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 28053504 bytes 2024-09-13T09:38:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 115159040 bytes 2024-09-13T09:38:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 201065472 bytes 2024-09-13T09:38:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 235917312 bytes 2024-09-13T09:38:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 239621120 bytes 2024-09-13T09:38:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 262491136 bytes 2024-09-13T09:38:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 332658688 bytes 2024-09-13T09:38:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 392304640 bytes 2024-09-13T09:38:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 429668352 bytes 2024-09-13T09:38:23.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 489637888 bytes 2024-09-13T09:38:23.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 510743552 bytes 2024-09-13T09:38:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 553081856 bytes 2024-09-13T09:38:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 643608576 bytes 2024-09-13T09:38:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 706783232 bytes 2024-09-13T09:38:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 724202496 bytes 2024-09-13T09:38:24.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 799933440 bytes 2024-09-13T09:38:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 823828480 bytes 2024-09-13T09:38:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 837969920 bytes 2024-09-13T09:38:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 902232064 bytes 2024-09-13T09:38:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 926453760 bytes 2024-09-13T09:38:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 986695680 bytes 2024-09-13T09:38:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1042873344 bytes 2024-09-13T09:38:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1140428800 bytes 2024-09-13T09:38:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1219999744 bytes 2024-09-13T09:38:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1287151616 bytes 2024-09-13T09:38:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1312578560 bytes 2024-09-13T09:38:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1407312896 bytes 2024-09-13T09:38:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1414695936 bytes 2024-09-13T09:38:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1415303168 bytes 2024-09-13T09:38:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1484597248 bytes 2024-09-13T09:38:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1519423488 bytes 2024-09-13T09:38:26.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1546523648 bytes 2024-09-13T09:38:26.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1629160448 bytes 2024-09-13T09:38:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1669311488 bytes 2024-09-13T09:38:26.503 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-13T09:38:26.503 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:38:26.504 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-13T09:38:26.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-13T09:38:26.504 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-13T09:38:26.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-13T09:38:26.507 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2024-09-13T09:38:26.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1756625920 bytes 2024-09-13T09:38:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1851616256 bytes 2024-09-13T09:38:26.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1862767616 bytes 2024-09-13T09:38:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1904435200 bytes 2024-09-13T09:38:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1945788416 bytes 2024-09-13T09:38:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 2011609088 bytes 2024-09-13T09:38:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 2071939072 bytes 2024-09-13T09:38:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 2137184256 bytes 2024-09-13T09:38:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 2183707648 bytes 2024-09-13T09:38:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2243049472 bytes 2024-09-13T09:38:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2246538240 bytes 2024-09-13T09:38:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2295270400 bytes 2024-09-13T09:38:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2313088000 bytes 2024-09-13T09:38:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2364211200 bytes 2024-09-13T09:38:28.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2398434304 bytes 2024-09-13T09:38:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2401538048 bytes 2024-09-13T09:38:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2485043200 bytes 2024-09-13T09:38:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2579189760 bytes 2024-09-13T09:38:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2579783680 bytes 2024-09-13T09:38:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2644502528 bytes 2024-09-13T09:38:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2749896704 bytes 2024-09-13T09:38:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2787644416 bytes 2024-09-13T09:38:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:38:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:38:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:38:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:38:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:38:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_16 2024-09-13T09:38:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2024-09-13T09:38:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:38:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:38:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:38:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_16 128M 2024-09-13T09:38:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:38:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:38:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:38:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:38:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_16 2024-09-13T09:38:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:38:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2024-09-13T09:38:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:38:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:38:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2024-09-13T09:38:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:38:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2024-09-13T09:38:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_16': 2024-09-13T09:38:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:38:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:38:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:38:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 131f9dc45460 2024-09-13T09:38:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.131f9dc45460 2024-09-13T09:38:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:38:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:38:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:38:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:38:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:38:29 2024 2024-09-13T09:38:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:38:29 2024 2024-09-13T09:38:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:38:29 2024 2024-09-13T09:38:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:38:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:38:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: cd59b635-c822-4dda-a6e6-0e788a2c4725 2024-09-13T09:38:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:38:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:38:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_16 2024-09-13T09:38:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:38:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:38:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=9 2024-09-13T09:38:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:38:30.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:38:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 22135808 bytes 2024-09-13T09:38:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 62420992 bytes 2024-09-13T09:38:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 85581824 bytes 2024-09-13T09:38:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 114601984 bytes 2024-09-13T09:38:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 174740480 bytes 2024-09-13T09:38:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 225955840 bytes 2024-09-13T09:38:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 226564096 bytes 2024-09-13T09:38:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 286543872 bytes 2024-09-13T09:38:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 326424576 bytes 2024-09-13T09:38:32.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 372022272 bytes 2024-09-13T09:38:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 399180800 bytes 2024-09-13T09:38:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 441633792 bytes 2024-09-13T09:38:32.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 481517568 bytes 2024-09-13T09:38:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 565836800 bytes 2024-09-13T09:38:32.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 626709504 bytes 2024-09-13T09:38:32.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 639589376 bytes 2024-09-13T09:38:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 667292672 bytes 2024-09-13T09:38:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 739531776 bytes 2024-09-13T09:38:33.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 868973568 bytes 2024-09-13T09:38:33.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 876534784 bytes 2024-09-13T09:38:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 894342144 bytes 2024-09-13T09:38:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 939036672 bytes 2024-09-13T09:38:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 967976960 bytes 2024-09-13T09:38:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1059291136 bytes 2024-09-13T09:38:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1128365056 bytes 2024-09-13T09:38:34.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1177058304 bytes 2024-09-13T09:38:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1211055104 bytes 2024-09-13T09:38:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1263463424 bytes 2024-09-13T09:38:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1275584512 bytes 2024-09-13T09:38:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1333793792 bytes 2024-09-13T09:38:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1398402048 bytes 2024-09-13T09:38:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1448094720 bytes 2024-09-13T09:38:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1531360256 bytes 2024-09-13T09:38:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1564344320 bytes 2024-09-13T09:38:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1624976384 bytes 2024-09-13T09:38:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1691681792 bytes 2024-09-13T09:38:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1743899648 bytes 2024-09-13T09:38:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1761509376 bytes 2024-09-13T09:38:35.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1789966336 bytes 2024-09-13T09:38:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1829787648 bytes 2024-09-13T09:38:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1907421184 bytes 2024-09-13T09:38:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1936488448 bytes 2024-09-13T09:38:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 2018479104 bytes 2024-09-13T09:38:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2045484032 bytes 2024-09-13T09:38:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2064022528 bytes 2024-09-13T09:38:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2085965824 bytes 2024-09-13T09:38:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2131744768 bytes 2024-09-13T09:38:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2170071040 bytes 2024-09-13T09:38:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2241856512 bytes 2024-09-13T09:38:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2271895552 bytes 2024-09-13T09:38:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2333306880 bytes 2024-09-13T09:38:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2382123008 bytes 2024-09-13T09:38:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2426083328 bytes 2024-09-13T09:38:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2436947968 bytes 2024-09-13T09:38:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2459260928 bytes 2024-09-13T09:38:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2488159232 bytes 2024-09-13T09:38:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2530882560 bytes 2024-09-13T09:38:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2550730752 bytes 2024-09-13T09:38:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2654496768 bytes 2024-09-13T09:38:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2717983744 bytes 2024-09-13T09:38:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2798966784 bytes 2024-09-13T09:38:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2821950464 bytes 2024-09-13T09:38:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2865233920 bytes 2024-09-13T09:38:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2970784768 bytes 2024-09-13T09:38:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 3026558976 bytes 2024-09-13T09:38:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 3098422272 bytes 2024-09-13T09:38:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 3115632640 bytes 2024-09-13T09:38:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:38:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:38:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_17 2024-09-13T09:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2024-09-13T09:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_17 128M 2024-09-13T09:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_17 2024-09-13T09:38:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:38:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2024-09-13T09:38:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:38:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:38:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2024-09-13T09:38:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:38:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2024-09-13T09:38:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_17': 2024-09-13T09:38:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:38:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:38:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:38:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13291b70d89e 2024-09-13T09:38:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13291b70d89e 2024-09-13T09:38:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:38:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:38:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:38:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:38:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:38:39 2024 2024-09-13T09:38:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:38:39 2024 2024-09-13T09:38:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:38:39 2024 2024-09-13T09:38:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:38:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:38:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: fc6f16a3-057d-4588-8097-7af12a77aa44 2024-09-13T09:38:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:38:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:38:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_17 2024-09-13T09:38:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:38:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:38:40.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=6 2024-09-13T09:38:40.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:38:40.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:38:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 29314048 bytes 2024-09-13T09:38:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 89080832 bytes 2024-09-13T09:38:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 116210688 bytes 2024-09-13T09:38:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 191795200 bytes 2024-09-13T09:38:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 232687616 bytes 2024-09-13T09:38:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 257126400 bytes 2024-09-13T09:38:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 359780352 bytes 2024-09-13T09:38:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 380004352 bytes 2024-09-13T09:38:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 380659712 bytes 2024-09-13T09:38:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 417950720 bytes 2024-09-13T09:38:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 479993856 bytes 2024-09-13T09:38:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 550055936 bytes 2024-09-13T09:38:42.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 625635328 bytes 2024-09-13T09:38:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 682708992 bytes 2024-09-13T09:38:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 771332096 bytes 2024-09-13T09:38:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 863901696 bytes 2024-09-13T09:38:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 956015616 bytes 2024-09-13T09:38:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 999237632 bytes 2024-09-13T09:38:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 1042013184 bytes 2024-09-13T09:38:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 1115566080 bytes 2024-09-13T09:38:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1134757888 bytes 2024-09-13T09:38:43.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1174880256 bytes 2024-09-13T09:38:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1205404672 bytes 2024-09-13T09:38:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1243666432 bytes 2024-09-13T09:38:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1359844352 bytes 2024-09-13T09:38:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1363920896 bytes 2024-09-13T09:38:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1386338304 bytes 2024-09-13T09:38:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1445238784 bytes 2024-09-13T09:38:44.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1485128704 bytes 2024-09-13T09:38:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1529995264 bytes 2024-09-13T09:38:45.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1578176512 bytes 2024-09-13T09:38:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1623780352 bytes 2024-09-13T09:38:45.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1681101824 bytes 2024-09-13T09:38:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1752525824 bytes 2024-09-13T09:38:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1787610112 bytes 2024-09-13T09:38:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1847242752 bytes 2024-09-13T09:38:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1926269952 bytes 2024-09-13T09:38:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1941583872 bytes 2024-09-13T09:38:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1970460672 bytes 2024-09-13T09:38:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 2032656384 bytes 2024-09-13T09:38:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 2048241664 bytes 2024-09-13T09:38:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 2084788224 bytes 2024-09-13T09:38:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 2121121792 bytes 2024-09-13T09:38:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2142092288 bytes 2024-09-13T09:38:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:38:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:38:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:38:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:38:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:38:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_18 2024-09-13T09:38:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2024-09-13T09:38:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:38:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:38:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:38:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_18 128M 2024-09-13T09:38:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:38:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:38:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:38:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:38:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:38:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_18 2024-09-13T09:38:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:38:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2024-09-13T09:38:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:38:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:38:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2024-09-13T09:38:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:38:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2024-09-13T09:38:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_18': 2024-09-13T09:38:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:38:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:38:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:38:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1332b877d1dd 2024-09-13T09:38:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1332b877d1dd 2024-09-13T09:38:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:38:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:38:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:38:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:38:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:38:46 2024 2024-09-13T09:38:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:38:46 2024 2024-09-13T09:38:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:38:46 2024 2024-09-13T09:38:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:38:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:38:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: d7e04b0a-b54c-4532-b814-c9ae04053bfa 2024-09-13T09:38:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:38:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:38:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_18 2024-09-13T09:38:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:38:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:38:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:38:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:38:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=14 2024-09-13T09:38:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:38:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:38:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 78514176 bytes 2024-09-13T09:38:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 92055552 bytes 2024-09-13T09:38:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 153601024 bytes 2024-09-13T09:38:48.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 220465152 bytes 2024-09-13T09:38:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 231507968 bytes 2024-09-13T09:38:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 264320000 bytes 2024-09-13T09:38:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 286141440 bytes 2024-09-13T09:38:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 298636288 bytes 2024-09-13T09:38:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 362166272 bytes 2024-09-13T09:38:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 389836800 bytes 2024-09-13T09:38:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 430271488 bytes 2024-09-13T09:38:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 440506368 bytes 2024-09-13T09:38:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 536576000 bytes 2024-09-13T09:38:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 569723904 bytes 2024-09-13T09:38:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 619460608 bytes 2024-09-13T09:38:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 670881792 bytes 2024-09-13T09:38:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 725470208 bytes 2024-09-13T09:38:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 762907648 bytes 2024-09-13T09:38:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 796111872 bytes 2024-09-13T09:38:50.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 829241344 bytes 2024-09-13T09:38:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 898381824 bytes 2024-09-13T09:38:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 924666880 bytes 2024-09-13T09:38:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 953819136 bytes 2024-09-13T09:38:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1006286848 bytes 2024-09-13T09:38:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1084289024 bytes 2024-09-13T09:38:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1127983104 bytes 2024-09-13T09:38:51.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1137134592 bytes 2024-09-13T09:38:51.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1141922816 bytes 2024-09-13T09:38:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1182828544 bytes 2024-09-13T09:38:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1183438848 bytes 2024-09-13T09:38:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1215452160 bytes 2024-09-13T09:38:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1270107136 bytes 2024-09-13T09:38:51.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1311989760 bytes 2024-09-13T09:38:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1377628160 bytes 2024-09-13T09:38:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1384701952 bytes 2024-09-13T09:38:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1407109120 bytes 2024-09-13T09:38:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1437312000 bytes 2024-09-13T09:38:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1478306816 bytes 2024-09-13T09:38:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1508299776 bytes 2024-09-13T09:38:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1591198720 bytes 2024-09-13T09:38:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1624637440 bytes 2024-09-13T09:38:53.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1656235008 bytes 2024-09-13T09:38:53.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1704688640 bytes 2024-09-13T09:38:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1715415040 bytes 2024-09-13T09:38:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1744499712 bytes 2024-09-13T09:38:53.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1764825088 bytes 2024-09-13T09:38:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1843467264 bytes 2024-09-13T09:38:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1880352768 bytes 2024-09-13T09:38:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1935936512 bytes 2024-09-13T09:38:54.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2024380416 bytes 2024-09-13T09:38:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2047215616 bytes 2024-09-13T09:38:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2099815424 bytes 2024-09-13T09:38:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2149647360 bytes 2024-09-13T09:38:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2163968000 bytes 2024-09-13T09:38:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2219236352 bytes 2024-09-13T09:38:54.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2295235584 bytes 2024-09-13T09:38:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2316438528 bytes 2024-09-13T09:38:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2357821440 bytes 2024-09-13T09:38:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2390740992 bytes 2024-09-13T09:38:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2413078528 bytes 2024-09-13T09:38:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2428683264 bytes 2024-09-13T09:38:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2471354368 bytes 2024-09-13T09:38:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2518903808 bytes 2024-09-13T09:38:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2584205312 bytes 2024-09-13T09:38:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2690347008 bytes 2024-09-13T09:38:56.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2781886464 bytes 2024-09-13T09:38:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2876462080 bytes 2024-09-13T09:38:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2921339904 bytes 2024-09-13T09:38:56.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2982167552 bytes 2024-09-13T09:38:56.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 3078932480 bytes 2024-09-13T09:38:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 3120510976 bytes 2024-09-13T09:38:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 3121124352 bytes 2024-09-13T09:38:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 3176744960 bytes 2024-09-13T09:38:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3280363520 bytes 2024-09-13T09:38:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3325264896 bytes 2024-09-13T09:38:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3344333824 bytes 2024-09-13T09:38:57.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3374473216 bytes 2024-09-13T09:38:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3454109696 bytes 2024-09-13T09:38:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3454726144 bytes 2024-09-13T09:38:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3494147072 bytes 2024-09-13T09:38:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3522557952 bytes 2024-09-13T09:38:58.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3529669632 bytes 2024-09-13T09:38:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3551432704 bytes 2024-09-13T09:38:58.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3572535296 bytes 2024-09-13T09:38:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3620216832 bytes 2024-09-13T09:38:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3621914624 bytes 2024-09-13T09:38:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3676678144 bytes 2024-09-13T09:38:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3710211072 bytes 2024-09-13T09:38:58.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3710810112 bytes 2024-09-13T09:38:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 3749510144 bytes 2024-09-13T09:38:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 3785363456 bytes 2024-09-13T09:38:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 3789401088 bytes 2024-09-13T09:38:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 3815325696 bytes 2024-09-13T09:38:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9400 IOs, 3882561536 bytes 2024-09-13T09:38:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9500 IOs, 3893997568 bytes 2024-09-13T09:38:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9600 IOs, 4020273152 bytes 2024-09-13T09:38:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9700 IOs, 4025071616 bytes 2024-09-13T09:39:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9800 IOs, 4064312320 bytes 2024-09-13T09:39:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9900 IOs, 4113283072 bytes 2024-09-13T09:39:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10000 IOs, 4146686976 bytes 2024-09-13T09:39:00.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10100 IOs, 4161672192 bytes 2024-09-13T09:39:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10200 IOs, 4203501568 bytes 2024-09-13T09:39:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10300 IOs, 4283572224 bytes 2024-09-13T09:39:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10400 IOs, 4305162240 bytes 2024-09-13T09:39:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10500 IOs, 4340955136 bytes 2024-09-13T09:39:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10600 IOs, 4404703232 bytes 2024-09-13T09:39:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10700 IOs, 4427076608 bytes 2024-09-13T09:39:01.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10800 IOs, 4523033600 bytes 2024-09-13T09:39:01.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10900 IOs, 4525687808 bytes 2024-09-13T09:39:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11000 IOs, 4573204480 bytes 2024-09-13T09:39:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11100 IOs, 4587026432 bytes 2024-09-13T09:39:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:39:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:39:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:39:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:39:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:39:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_19 2024-09-13T09:39:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2024-09-13T09:39:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:39:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:39:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:39:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:39:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:39:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_19 128M 2024-09-13T09:39:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:39:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:39:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:39:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:39:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:39:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:39:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_19 2024-09-13T09:39:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:39:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2024-09-13T09:39:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:39:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:39:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:39:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:39:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2024-09-13T09:39:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:39:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2024-09-13T09:39:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_19': 2024-09-13T09:39:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:39:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:39:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:39:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 133b31ddbeab 2024-09-13T09:39:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.133b31ddbeab 2024-09-13T09:39:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:39:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:39:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:39:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:39:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:39:01 2024 2024-09-13T09:39:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:39:01 2024 2024-09-13T09:39:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:39:01 2024 2024-09-13T09:39:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:39:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:39:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 98a2a84e-2461-434e-b3e3-a5de2949eb05 2024-09-13T09:39:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:39:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:39:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_19 2024-09-13T09:39:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:39:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:39:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:39:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:39:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=13 2024-09-13T09:39:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:39:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:39:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 4861952 bytes 2024-09-13T09:39:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 23392256 bytes 2024-09-13T09:39:03.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 84583424 bytes 2024-09-13T09:39:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 124399616 bytes 2024-09-13T09:39:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 152043520 bytes 2024-09-13T09:39:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 226126848 bytes 2024-09-13T09:39:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 248811520 bytes 2024-09-13T09:39:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 260708352 bytes 2024-09-13T09:39:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 326278144 bytes 2024-09-13T09:39:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 346365952 bytes 2024-09-13T09:39:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 418016256 bytes 2024-09-13T09:39:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 434636800 bytes 2024-09-13T09:39:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 493697024 bytes 2024-09-13T09:39:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 503915520 bytes 2024-09-13T09:39:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 524894208 bytes 2024-09-13T09:39:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 605972480 bytes 2024-09-13T09:39:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 645566464 bytes 2024-09-13T09:39:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 663715840 bytes 2024-09-13T09:39:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 668588032 bytes 2024-09-13T09:39:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 711026688 bytes 2024-09-13T09:39:05.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 712923136 bytes 2024-09-13T09:39:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 713501696 bytes 2024-09-13T09:39:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 792185856 bytes 2024-09-13T09:39:05.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 801062912 bytes 2024-09-13T09:39:05.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 834456576 bytes 2024-09-13T09:39:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 899987456 bytes 2024-09-13T09:39:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 900606976 bytes 2024-09-13T09:39:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 922352640 bytes 2024-09-13T09:39:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 950171648 bytes 2024-09-13T09:39:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1022851072 bytes 2024-09-13T09:39:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1091992576 bytes 2024-09-13T09:39:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1117196288 bytes 2024-09-13T09:39:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1129273344 bytes 2024-09-13T09:39:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1212815360 bytes 2024-09-13T09:39:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1246892032 bytes 2024-09-13T09:39:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1263443968 bytes 2024-09-13T09:39:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1288690688 bytes 2024-09-13T09:39:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1300551680 bytes 2024-09-13T09:39:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1350966272 bytes 2024-09-13T09:39:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1424315392 bytes 2024-09-13T09:39:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1447053312 bytes 2024-09-13T09:39:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1493319680 bytes 2024-09-13T09:39:08.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1527462912 bytes 2024-09-13T09:39:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1560581120 bytes 2024-09-13T09:39:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1602307072 bytes 2024-09-13T09:39:08.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1611146240 bytes 2024-09-13T09:39:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1611770880 bytes 2024-09-13T09:39:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1619542016 bytes 2024-09-13T09:39:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1636240384 bytes 2024-09-13T09:39:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1656634368 bytes 2024-09-13T09:39:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 1737482240 bytes 2024-09-13T09:39:08.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 1771408384 bytes 2024-09-13T09:39:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 1844536320 bytes 2024-09-13T09:39:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 1857765376 bytes 2024-09-13T09:39:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 1926605824 bytes 2024-09-13T09:39:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 1928291328 bytes 2024-09-13T09:39:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 1979945984 bytes 2024-09-13T09:39:09.757 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2024-09-13T09:39:09.755+0000 7ff85b706640 -1 reset not still connected to 0x5568efdf4750 2024-09-13T09:39:09.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2057003008 bytes 2024-09-13T09:39:09.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2115121152 bytes 2024-09-13T09:39:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2195405824 bytes 2024-09-13T09:39:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2210772992 bytes 2024-09-13T09:39:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2265684992 bytes 2024-09-13T09:39:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2276716544 bytes 2024-09-13T09:39:10.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2369850368 bytes 2024-09-13T09:39:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2391766016 bytes 2024-09-13T09:39:10.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2469617664 bytes 2024-09-13T09:39:10.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2470224896 bytes 2024-09-13T09:39:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2530593792 bytes 2024-09-13T09:39:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2563539968 bytes 2024-09-13T09:39:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2584028160 bytes 2024-09-13T09:39:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2639236096 bytes 2024-09-13T09:39:11.395 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6525191821825409) > weight (0.25) 2024-09-13T09:39:11.395 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.509681230882822) > weight (0.25) 2024-09-13T09:39:11.395 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.45807825743418007) > weight (0.25) 2024-09-13T09:39:11.395 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.41607812250410225) > weight (0.25) 2024-09-13T09:39:11.395 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2024-09-13T09:39:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2661499904 bytes 2024-09-13T09:39:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2716372992 bytes 2024-09-13T09:39:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 2805783552 bytes 2024-09-13T09:39:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 2885950464 bytes 2024-09-13T09:39:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 2902705152 bytes 2024-09-13T09:39:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 2903340032 bytes 2024-09-13T09:39:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 2921352192 bytes 2024-09-13T09:39:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 2956571648 bytes 2024-09-13T09:39:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 2998876160 bytes 2024-09-13T09:39:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3069972480 bytes 2024-09-13T09:39:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3095560192 bytes 2024-09-13T09:39:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3150525440 bytes 2024-09-13T09:39:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3191410688 bytes 2024-09-13T09:39:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3242042368 bytes 2024-09-13T09:39:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3291144192 bytes 2024-09-13T09:39:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3355636736 bytes 2024-09-13T09:39:13.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3368890368 bytes 2024-09-13T09:39:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3420404736 bytes 2024-09-13T09:39:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 3422448640 bytes 2024-09-13T09:39:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 3520048128 bytes 2024-09-13T09:39:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 3568820224 bytes 2024-09-13T09:39:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 3616513024 bytes 2024-09-13T09:39:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9400 IOs, 3677873152 bytes 2024-09-13T09:39:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9500 IOs, 3678498816 bytes 2024-09-13T09:39:14.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9600 IOs, 3761308672 bytes 2024-09-13T09:39:14.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9700 IOs, 3820989440 bytes 2024-09-13T09:39:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9800 IOs, 3858575360 bytes 2024-09-13T09:39:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9900 IOs, 3903164416 bytes 2024-09-13T09:39:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10000 IOs, 3932954624 bytes 2024-09-13T09:39:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10100 IOs, 3941078016 bytes 2024-09-13T09:39:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10200 IOs, 3983325184 bytes 2024-09-13T09:39:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10300 IOs, 4033688576 bytes 2024-09-13T09:39:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10400 IOs, 4062201856 bytes 2024-09-13T09:39:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10500 IOs, 4094404608 bytes 2024-09-13T09:39:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10600 IOs, 4137174016 bytes 2024-09-13T09:39:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10700 IOs, 4215066624 bytes 2024-09-13T09:39:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10800 IOs, 4216375296 bytes 2024-09-13T09:39:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:39:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:39:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:39:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:39:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:39:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_20 2024-09-13T09:39:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2024-09-13T09:39:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:39:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:39:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:39:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:39:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:39:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_20 128M 2024-09-13T09:39:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:39:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:39:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:39:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:39:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:39:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:39:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_20 2024-09-13T09:39:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:39:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2024-09-13T09:39:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:39:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:39:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:39:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:39:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2024-09-13T09:39:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:39:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2024-09-13T09:39:16.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_20': 2024-09-13T09:39:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:39:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:39:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:39:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1343816ccb0e 2024-09-13T09:39:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1343816ccb0e 2024-09-13T09:39:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:39:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:39:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:39:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:39:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:39:16 2024 2024-09-13T09:39:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:39:16 2024 2024-09-13T09:39:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:39:16 2024 2024-09-13T09:39:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:39:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:39:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: ab0efa63-cc31-4345-9caf-c5a58d606330 2024-09-13T09:39:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:39:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:39:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_20 2024-09-13T09:39:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:39:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:39:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:39:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:39:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=9 2024-09-13T09:39:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:39:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:39:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 20478976 bytes 2024-09-13T09:39:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 38007808 bytes 2024-09-13T09:39:17.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 70117376 bytes 2024-09-13T09:39:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 95188992 bytes 2024-09-13T09:39:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 100286464 bytes 2024-09-13T09:39:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 140598272 bytes 2024-09-13T09:39:17.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 164894720 bytes 2024-09-13T09:39:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 184526848 bytes 2024-09-13T09:39:17.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 239180800 bytes 2024-09-13T09:39:18.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 262982656 bytes 2024-09-13T09:39:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 330702848 bytes 2024-09-13T09:39:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 381784064 bytes 2024-09-13T09:39:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 382381056 bytes 2024-09-13T09:39:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 416185344 bytes 2024-09-13T09:39:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 469644288 bytes 2024-09-13T09:39:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 559293440 bytes 2024-09-13T09:39:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 593522688 bytes 2024-09-13T09:39:19.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 609494016 bytes 2024-09-13T09:39:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 660669440 bytes 2024-09-13T09:39:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 686455808 bytes 2024-09-13T09:39:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 749048832 bytes 2024-09-13T09:39:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 806309888 bytes 2024-09-13T09:39:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 830645248 bytes 2024-09-13T09:39:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 867265536 bytes 2024-09-13T09:39:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 899519488 bytes 2024-09-13T09:39:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 972891136 bytes 2024-09-13T09:39:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1007056896 bytes 2024-09-13T09:39:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1013801984 bytes 2024-09-13T09:39:20.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1033924608 bytes 2024-09-13T09:39:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1114117120 bytes 2024-09-13T09:39:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1128646656 bytes 2024-09-13T09:39:20.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1141592064 bytes 2024-09-13T09:39:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1204035584 bytes 2024-09-13T09:39:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1211800576 bytes 2024-09-13T09:39:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1250918400 bytes 2024-09-13T09:39:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1273736192 bytes 2024-09-13T09:39:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1299067904 bytes 2024-09-13T09:39:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1344181248 bytes 2024-09-13T09:39:21.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1362712576 bytes 2024-09-13T09:39:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1396033536 bytes 2024-09-13T09:39:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1411346432 bytes 2024-09-13T09:39:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1439381504 bytes 2024-09-13T09:39:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1439990784 bytes 2024-09-13T09:39:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1452776448 bytes 2024-09-13T09:39:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1487527936 bytes 2024-09-13T09:39:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1559749632 bytes 2024-09-13T09:39:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1566682112 bytes 2024-09-13T09:39:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1601935360 bytes 2024-09-13T09:39:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1625449472 bytes 2024-09-13T09:39:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1674847232 bytes 2024-09-13T09:39:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 1720216576 bytes 2024-09-13T09:39:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 1752646656 bytes 2024-09-13T09:39:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 1850097664 bytes 2024-09-13T09:39:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 1936736256 bytes 2024-09-13T09:39:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 1990240256 bytes 2024-09-13T09:39:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2015566848 bytes 2024-09-13T09:39:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2074966016 bytes 2024-09-13T09:39:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2146076672 bytes 2024-09-13T09:39:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2202252288 bytes 2024-09-13T09:39:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2263830528 bytes 2024-09-13T09:39:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2322521088 bytes 2024-09-13T09:39:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2352855040 bytes 2024-09-13T09:39:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2389449728 bytes 2024-09-13T09:39:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2404593664 bytes 2024-09-13T09:39:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2449759232 bytes 2024-09-13T09:39:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2498045952 bytes 2024-09-13T09:39:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2540993536 bytes 2024-09-13T09:39:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2615608320 bytes 2024-09-13T09:39:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2644052992 bytes 2024-09-13T09:39:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2671597568 bytes 2024-09-13T09:39:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2687092736 bytes 2024-09-13T09:39:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:39:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:39:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:39:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:39:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:39:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_21 2024-09-13T09:39:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2024-09-13T09:39:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:39:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:39:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:39:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:39:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:39:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_21 128M 2024-09-13T09:39:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:39:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:39:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:39:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:39:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:39:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:39:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_21 2024-09-13T09:39:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:39:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2024-09-13T09:39:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:39:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:39:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:39:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:39:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2024-09-13T09:39:26.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:39:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2024-09-13T09:39:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_21': 2024-09-13T09:39:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:39:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:39:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:39:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 134b8d098472 2024-09-13T09:39:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.134b8d098472 2024-09-13T09:39:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:39:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:39:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:39:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:39:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:39:25 2024 2024-09-13T09:39:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:39:25 2024 2024-09-13T09:39:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:39:25 2024 2024-09-13T09:39:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:39:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:39:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 47058edf-e75b-483b-aaa7-812779184a08 2024-09-13T09:39:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:39:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:39:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_21 2024-09-13T09:39:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:39:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:39:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:39:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:39:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=11 2024-09-13T09:39:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:39:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:39:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 21896192 bytes 2024-09-13T09:39:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 56301568 bytes 2024-09-13T09:39:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 100678656 bytes 2024-09-13T09:39:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 155102208 bytes 2024-09-13T09:39:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 259861504 bytes 2024-09-13T09:39:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 294138880 bytes 2024-09-13T09:39:27.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 345744384 bytes 2024-09-13T09:39:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 390994944 bytes 2024-09-13T09:39:28.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 427301888 bytes 2024-09-13T09:39:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 470352896 bytes 2024-09-13T09:39:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 561274880 bytes 2024-09-13T09:39:28.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 562475008 bytes 2024-09-13T09:39:28.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 653578240 bytes 2024-09-13T09:39:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 671895552 bytes 2024-09-13T09:39:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 708988928 bytes 2024-09-13T09:39:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 763571200 bytes 2024-09-13T09:39:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 802840576 bytes 2024-09-13T09:39:29.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 828413952 bytes 2024-09-13T09:39:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 912075776 bytes 2024-09-13T09:39:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 937734144 bytes 2024-09-13T09:39:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1003980800 bytes 2024-09-13T09:39:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1063822336 bytes 2024-09-13T09:39:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1084935168 bytes 2024-09-13T09:39:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1152080896 bytes 2024-09-13T09:39:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1202099200 bytes 2024-09-13T09:39:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1234627584 bytes 2024-09-13T09:39:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1345872896 bytes 2024-09-13T09:39:30.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1448022016 bytes 2024-09-13T09:39:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1515877376 bytes 2024-09-13T09:39:31.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1589019648 bytes 2024-09-13T09:39:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1594198016 bytes 2024-09-13T09:39:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1650201600 bytes 2024-09-13T09:39:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1717677056 bytes 2024-09-13T09:39:31.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1803961344 bytes 2024-09-13T09:39:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1849754624 bytes 2024-09-13T09:39:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1892294656 bytes 2024-09-13T09:39:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1945288704 bytes 2024-09-13T09:39:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1982518272 bytes 2024-09-13T09:39:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 2044947456 bytes 2024-09-13T09:39:32.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 2082886656 bytes 2024-09-13T09:39:32.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 2092410880 bytes 2024-09-13T09:39:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 2132098048 bytes 2024-09-13T09:39:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 2162673664 bytes 2024-09-13T09:39:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2179105792 bytes 2024-09-13T09:39:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2228854784 bytes 2024-09-13T09:39:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2282648576 bytes 2024-09-13T09:39:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2304050176 bytes 2024-09-13T09:39:33.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2382115840 bytes 2024-09-13T09:39:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2385747968 bytes 2024-09-13T09:39:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2435449856 bytes 2024-09-13T09:39:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2444518400 bytes 2024-09-13T09:39:33.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2509020160 bytes 2024-09-13T09:39:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2572569600 bytes 2024-09-13T09:39:34.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2661761024 bytes 2024-09-13T09:39:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2707422208 bytes 2024-09-13T09:39:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2739651584 bytes 2024-09-13T09:39:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2762694656 bytes 2024-09-13T09:39:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2839528448 bytes 2024-09-13T09:39:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2911463424 bytes 2024-09-13T09:39:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2938370048 bytes 2024-09-13T09:39:34.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2991291392 bytes 2024-09-13T09:39:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 3054243840 bytes 2024-09-13T09:39:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 3135989760 bytes 2024-09-13T09:39:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 3187289088 bytes 2024-09-13T09:39:35.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 3227614208 bytes 2024-09-13T09:39:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 3262272512 bytes 2024-09-13T09:39:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 3280479232 bytes 2024-09-13T09:39:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 3309648896 bytes 2024-09-13T09:39:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 3389955072 bytes 2024-09-13T09:39:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 3427827712 bytes 2024-09-13T09:39:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 3446857728 bytes 2024-09-13T09:39:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 3477940224 bytes 2024-09-13T09:39:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 3567859712 bytes 2024-09-13T09:39:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3614565376 bytes 2024-09-13T09:39:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3673246720 bytes 2024-09-13T09:39:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3705160704 bytes 2024-09-13T09:39:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3751348224 bytes 2024-09-13T09:39:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3798654976 bytes 2024-09-13T09:39:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3799265280 bytes 2024-09-13T09:39:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3864603648 bytes 2024-09-13T09:39:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3902789632 bytes 2024-09-13T09:39:37.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3916297216 bytes 2024-09-13T09:39:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3952309248 bytes 2024-09-13T09:39:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3968858112 bytes 2024-09-13T09:39:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:39:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:39:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:39:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:39:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:39:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_22 2024-09-13T09:39:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2024-09-13T09:39:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:39:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:39:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:39:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:39:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:39:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_22 128M 2024-09-13T09:39:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:39:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:39:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:39:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:39:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:39:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:39:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_22 2024-09-13T09:39:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:39:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2024-09-13T09:39:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:39:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:39:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:39:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:39:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2024-09-13T09:39:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:39:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2024-09-13T09:39:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_22': 2024-09-13T09:39:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:39:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:39:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:39:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1354bae850ab 2024-09-13T09:39:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1354bae850ab 2024-09-13T09:39:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:39:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:39:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:39:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:39:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:39:37 2024 2024-09-13T09:39:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:39:37 2024 2024-09-13T09:39:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:39:37 2024 2024-09-13T09:39:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:39:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:39:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 9cf71ada-0f10-4f47-91dd-9b7f8293279d 2024-09-13T09:39:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:39:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:39:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_22 2024-09-13T09:39:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:39:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:39:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:39:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:39:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2024-09-13T09:39:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:39:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:39:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 37959680 bytes 2024-09-13T09:39:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 44235776 bytes 2024-09-13T09:39:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 57988096 bytes 2024-09-13T09:39:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 78720000 bytes 2024-09-13T09:39:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 159268864 bytes 2024-09-13T09:39:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 166537216 bytes 2024-09-13T09:39:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 191182848 bytes 2024-09-13T09:39:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 202575872 bytes 2024-09-13T09:39:39.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 252162048 bytes 2024-09-13T09:39:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 254001152 bytes 2024-09-13T09:39:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 295150592 bytes 2024-09-13T09:39:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 319226880 bytes 2024-09-13T09:39:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 344716288 bytes 2024-09-13T09:39:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 347548672 bytes 2024-09-13T09:39:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 407538688 bytes 2024-09-13T09:39:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 445112320 bytes 2024-09-13T09:39:40.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 453321728 bytes 2024-09-13T09:39:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 483230720 bytes 2024-09-13T09:39:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 599527424 bytes 2024-09-13T09:39:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 649952256 bytes 2024-09-13T09:39:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 696027136 bytes 2024-09-13T09:39:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 823388160 bytes 2024-09-13T09:39:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 841864192 bytes 2024-09-13T09:39:41.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 865448960 bytes 2024-09-13T09:39:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 937413632 bytes 2024-09-13T09:39:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 972123136 bytes 2024-09-13T09:39:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 995757056 bytes 2024-09-13T09:39:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1014908928 bytes 2024-09-13T09:39:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1090838528 bytes 2024-09-13T09:39:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1163262976 bytes 2024-09-13T09:39:41.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1173147648 bytes 2024-09-13T09:39:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1190435840 bytes 2024-09-13T09:39:42.108 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2024-09-13T09:39:42.106+0000 7f3610499640 -1 reset not still connected to 0x55cdec253e10 2024-09-13T09:39:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1207620608 bytes 2024-09-13T09:39:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1273247744 bytes 2024-09-13T09:39:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1292608512 bytes 2024-09-13T09:39:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1334517760 bytes 2024-09-13T09:39:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1412582400 bytes 2024-09-13T09:39:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1432601600 bytes 2024-09-13T09:39:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1475690496 bytes 2024-09-13T09:39:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1525794816 bytes 2024-09-13T09:39:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1526422528 bytes 2024-09-13T09:39:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1550345216 bytes 2024-09-13T09:39:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1578821632 bytes 2024-09-13T09:39:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1621417984 bytes 2024-09-13T09:39:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1658685440 bytes 2024-09-13T09:39:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1702496256 bytes 2024-09-13T09:39:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1720025088 bytes 2024-09-13T09:39:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1771557888 bytes 2024-09-13T09:39:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1824124928 bytes 2024-09-13T09:39:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1847727104 bytes 2024-09-13T09:39:44.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 1884498944 bytes 2024-09-13T09:39:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 1943256064 bytes 2024-09-13T09:39:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2010868736 bytes 2024-09-13T09:39:44.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2064984064 bytes 2024-09-13T09:39:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2069006336 bytes 2024-09-13T09:39:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2118731776 bytes 2024-09-13T09:39:45.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2222147584 bytes 2024-09-13T09:39:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2258130944 bytes 2024-09-13T09:39:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2284568576 bytes 2024-09-13T09:39:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2328808448 bytes 2024-09-13T09:39:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2443172864 bytes 2024-09-13T09:39:45.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2467101696 bytes 2024-09-13T09:39:45.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2506364928 bytes 2024-09-13T09:39:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2553373696 bytes 2024-09-13T09:39:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2609518592 bytes 2024-09-13T09:39:46.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2610114560 bytes 2024-09-13T09:39:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2610702336 bytes 2024-09-13T09:39:46.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2662934528 bytes 2024-09-13T09:39:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2681520128 bytes 2024-09-13T09:39:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2722665472 bytes 2024-09-13T09:39:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2774814720 bytes 2024-09-13T09:39:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2793971712 bytes 2024-09-13T09:39:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2822046720 bytes 2024-09-13T09:39:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 2835895296 bytes 2024-09-13T09:39:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 2862010368 bytes 2024-09-13T09:39:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3010380800 bytes 2024-09-13T09:39:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3030349824 bytes 2024-09-13T09:39:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3062896640 bytes 2024-09-13T09:39:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3092081664 bytes 2024-09-13T09:39:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3113081856 bytes 2024-09-13T09:39:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3140175872 bytes 2024-09-13T09:39:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3174257664 bytes 2024-09-13T09:39:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3227750400 bytes 2024-09-13T09:39:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3273491456 bytes 2024-09-13T09:39:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3321177088 bytes 2024-09-13T09:39:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3383065600 bytes 2024-09-13T09:39:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3455114240 bytes 2024-09-13T09:39:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3535387648 bytes 2024-09-13T09:39:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3582332928 bytes 2024-09-13T09:39:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 3657927680 bytes 2024-09-13T09:39:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 3748261888 bytes 2024-09-13T09:39:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 3785732096 bytes 2024-09-13T09:39:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 3786352640 bytes 2024-09-13T09:39:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9400 IOs, 3810099200 bytes 2024-09-13T09:39:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9500 IOs, 3827856384 bytes 2024-09-13T09:39:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9600 IOs, 3855246336 bytes 2024-09-13T09:39:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:39:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:39:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:39:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:39:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:39:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_23 2024-09-13T09:39:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2024-09-13T09:39:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:39:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:39:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:39:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:39:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:39:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_23 128M 2024-09-13T09:39:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:39:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:39:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:39:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:39:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:39:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:39:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:39:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_23 2024-09-13T09:39:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:39:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2024-09-13T09:39:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:39:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:39:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:39:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:39:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2024-09-13T09:39:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:39:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2024-09-13T09:39:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_23': 2024-09-13T09:39:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:39:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:39:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:39:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 135d3c42cb34 2024-09-13T09:39:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.135d3c42cb34 2024-09-13T09:39:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:39:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:39:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:39:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:39:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:39:50 2024 2024-09-13T09:39:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:39:50 2024 2024-09-13T09:39:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:39:50 2024 2024-09-13T09:39:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:39:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:39:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: b1f94b49-8135-472a-8920-462d2d90e565 2024-09-13T09:39:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:39:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:39:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_23 2024-09-13T09:39:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:39:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:39:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:39:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:39:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2024-09-13T09:39:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:39:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:39:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 18333696 bytes 2024-09-13T09:39:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 55467008 bytes 2024-09-13T09:39:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 66805760 bytes 2024-09-13T09:39:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 98120704 bytes 2024-09-13T09:39:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 111463424 bytes 2024-09-13T09:39:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 193589248 bytes 2024-09-13T09:39:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 246951936 bytes 2024-09-13T09:39:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 311099392 bytes 2024-09-13T09:39:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 377986048 bytes 2024-09-13T09:39:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 396096512 bytes 2024-09-13T09:39:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 396706816 bytes 2024-09-13T09:39:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 430811136 bytes 2024-09-13T09:39:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 445858816 bytes 2024-09-13T09:39:52.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 503541760 bytes 2024-09-13T09:39:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 538992640 bytes 2024-09-13T09:39:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 546583552 bytes 2024-09-13T09:39:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 581197824 bytes 2024-09-13T09:39:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 624854016 bytes 2024-09-13T09:39:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 707609600 bytes 2024-09-13T09:39:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 750604288 bytes 2024-09-13T09:39:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 790458368 bytes 2024-09-13T09:39:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 838758400 bytes 2024-09-13T09:39:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 866660352 bytes 2024-09-13T09:39:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 939868160 bytes 2024-09-13T09:39:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 940487680 bytes 2024-09-13T09:39:54.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 964064256 bytes 2024-09-13T09:39:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1039148032 bytes 2024-09-13T09:39:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1078081536 bytes 2024-09-13T09:39:54.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1126305792 bytes 2024-09-13T09:39:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1133925376 bytes 2024-09-13T09:39:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1143861248 bytes 2024-09-13T09:39:55.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1165593600 bytes 2024-09-13T09:39:55.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1171619840 bytes 2024-09-13T09:39:55.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1197256704 bytes 2024-09-13T09:39:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1252002816 bytes 2024-09-13T09:39:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1321095168 bytes 2024-09-13T09:39:55.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1358433280 bytes 2024-09-13T09:39:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1359057920 bytes 2024-09-13T09:39:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1390193664 bytes 2024-09-13T09:39:56.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1390832640 bytes 2024-09-13T09:39:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1486843904 bytes 2024-09-13T09:39:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1564967936 bytes 2024-09-13T09:39:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1592968192 bytes 2024-09-13T09:39:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1617496064 bytes 2024-09-13T09:39:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1622810624 bytes 2024-09-13T09:39:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1665741824 bytes 2024-09-13T09:39:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1749297152 bytes 2024-09-13T09:39:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1757443072 bytes 2024-09-13T09:39:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1817649152 bytes 2024-09-13T09:39:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1893977088 bytes 2024-09-13T09:39:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 1930813440 bytes 2024-09-13T09:39:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 1973113856 bytes 2024-09-13T09:39:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2009105408 bytes 2024-09-13T09:39:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2086318080 bytes 2024-09-13T09:39:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2123362304 bytes 2024-09-13T09:39:58.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2161060864 bytes 2024-09-13T09:39:58.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2207495168 bytes 2024-09-13T09:39:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2255218688 bytes 2024-09-13T09:39:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2257431552 bytes 2024-09-13T09:39:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2347372544 bytes 2024-09-13T09:39:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2402952192 bytes 2024-09-13T09:39:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2428013568 bytes 2024-09-13T09:39:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2480727040 bytes 2024-09-13T09:39:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2525433856 bytes 2024-09-13T09:39:59.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2558096384 bytes 2024-09-13T09:39:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2566109184 bytes 2024-09-13T09:39:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2644859904 bytes 2024-09-13T09:39:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2692994048 bytes 2024-09-13T09:39:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2693608448 bytes 2024-09-13T09:39:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2696945664 bytes 2024-09-13T09:40:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2756681728 bytes 2024-09-13T09:40:00.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2782584832 bytes 2024-09-13T09:40:00.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2836634624 bytes 2024-09-13T09:40:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 2849405952 bytes 2024-09-13T09:40:00.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 2904017920 bytes 2024-09-13T09:40:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 2916620288 bytes 2024-09-13T09:40:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 2970594304 bytes 2024-09-13T09:40:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3048682496 bytes 2024-09-13T09:40:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3158532096 bytes 2024-09-13T09:40:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3210979328 bytes 2024-09-13T09:40:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3276870656 bytes 2024-09-13T09:40:01.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3322254336 bytes 2024-09-13T09:40:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3395006464 bytes 2024-09-13T09:40:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3456765952 bytes 2024-09-13T09:40:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3463041024 bytes 2024-09-13T09:40:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3463672832 bytes 2024-09-13T09:40:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3530389504 bytes 2024-09-13T09:40:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3531010048 bytes 2024-09-13T09:40:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3563134976 bytes 2024-09-13T09:40:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 3630477312 bytes 2024-09-13T09:40:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 3712262144 bytes 2024-09-13T09:40:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 3741090816 bytes 2024-09-13T09:40:03.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:40:03.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:40:03.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:40:03.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:40:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:40:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_24 2024-09-13T09:40:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2024-09-13T09:40:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:40:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:40:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:40:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_24 128M 2024-09-13T09:40:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:40:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:40:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:40:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:40:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_24 2024-09-13T09:40:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:40:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2024-09-13T09:40:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:40:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:40:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2024-09-13T09:40:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:40:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2024-09-13T09:40:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_24': 2024-09-13T09:40:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:40:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:40:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:40:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13653db76450 2024-09-13T09:40:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13653db76450 2024-09-13T09:40:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:40:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:40:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:40:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:40:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:40:03 2024 2024-09-13T09:40:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:40:03 2024 2024-09-13T09:40:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:40:03 2024 2024-09-13T09:40:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:40:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:40:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: aa7749ba-cd93-4a24-906c-a357a64f71e8 2024-09-13T09:40:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:40:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:40:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_24 2024-09-13T09:40:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:40:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:40:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=7 2024-09-13T09:40:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:40:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:40:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 70327296 bytes 2024-09-13T09:40:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 95951872 bytes 2024-09-13T09:40:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 123369472 bytes 2024-09-13T09:40:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 180627456 bytes 2024-09-13T09:40:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 220474368 bytes 2024-09-13T09:40:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 243921920 bytes 2024-09-13T09:40:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 326111232 bytes 2024-09-13T09:40:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 363156480 bytes 2024-09-13T09:40:05.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 464844800 bytes 2024-09-13T09:40:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 487316480 bytes 2024-09-13T09:40:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 554299392 bytes 2024-09-13T09:40:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 615767040 bytes 2024-09-13T09:40:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 636466176 bytes 2024-09-13T09:40:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 646648832 bytes 2024-09-13T09:40:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 668958720 bytes 2024-09-13T09:40:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 710556672 bytes 2024-09-13T09:40:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 730660864 bytes 2024-09-13T09:40:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 811600896 bytes 2024-09-13T09:40:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 839548928 bytes 2024-09-13T09:40:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 846739456 bytes 2024-09-13T09:40:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 904590336 bytes 2024-09-13T09:40:07.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 920396800 bytes 2024-09-13T09:40:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 964794368 bytes 2024-09-13T09:40:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1018634240 bytes 2024-09-13T09:40:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1056901120 bytes 2024-09-13T09:40:07.509 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-13T09:40:07.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-13T09:40:07.510 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-13T09:40:07.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T09:40:07.509+0000 7fa0e1c65640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 39852) UID: 1000 2024-09-13T09:40:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1107939328 bytes 2024-09-13T09:40:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1133110272 bytes 2024-09-13T09:40:07.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1191746560 bytes 2024-09-13T09:40:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1273755648 bytes 2024-09-13T09:40:08.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1314350080 bytes 2024-09-13T09:40:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1356769280 bytes 2024-09-13T09:40:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1369365504 bytes 2024-09-13T09:40:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1405082624 bytes 2024-09-13T09:40:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1461847040 bytes 2024-09-13T09:40:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1523950592 bytes 2024-09-13T09:40:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1562392576 bytes 2024-09-13T09:40:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1571668992 bytes 2024-09-13T09:40:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1619106816 bytes 2024-09-13T09:40:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1658921984 bytes 2024-09-13T09:40:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1682674688 bytes 2024-09-13T09:40:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1697814528 bytes 2024-09-13T09:40:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1698433024 bytes 2024-09-13T09:40:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1764293632 bytes 2024-09-13T09:40:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1779244032 bytes 2024-09-13T09:40:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1801284608 bytes 2024-09-13T09:40:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1863754752 bytes 2024-09-13T09:40:10.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1896938496 bytes 2024-09-13T09:40:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1967126528 bytes 2024-09-13T09:40:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2033932288 bytes 2024-09-13T09:40:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2084690944 bytes 2024-09-13T09:40:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2130376704 bytes 2024-09-13T09:40:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2150858752 bytes 2024-09-13T09:40:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2170218496 bytes 2024-09-13T09:40:11.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2176468992 bytes 2024-09-13T09:40:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2212573184 bytes 2024-09-13T09:40:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:40:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:40:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:40:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:40:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:40:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_25 2024-09-13T09:40:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2024-09-13T09:40:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:11.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:11.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:11.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:40:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:40:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:40:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_25 128M 2024-09-13T09:40:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:40:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:40:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:40:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:40:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_25 2024-09-13T09:40:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:40:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2024-09-13T09:40:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:40:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:40:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2024-09-13T09:40:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:40:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2024-09-13T09:40:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_25': 2024-09-13T09:40:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:40:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:40:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:40:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 136ec6a40292 2024-09-13T09:40:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.136ec6a40292 2024-09-13T09:40:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:40:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:40:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:40:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:40:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:40:11 2024 2024-09-13T09:40:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:40:11 2024 2024-09-13T09:40:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:40:11 2024 2024-09-13T09:40:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:40:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:40:12.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: b4563f80-e68a-4c65-bea2-f31514c46d7c 2024-09-13T09:40:12.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:40:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:40:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_25 2024-09-13T09:40:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:40:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:40:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=7 2024-09-13T09:40:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:40:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:40:12.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 31404032 bytes 2024-09-13T09:40:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 55531520 bytes 2024-09-13T09:40:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 103458816 bytes 2024-09-13T09:40:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 114828288 bytes 2024-09-13T09:40:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 183503872 bytes 2024-09-13T09:40:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 201932800 bytes 2024-09-13T09:40:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 266323968 bytes 2024-09-13T09:40:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 303344640 bytes 2024-09-13T09:40:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 396160000 bytes 2024-09-13T09:40:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 412642304 bytes 2024-09-13T09:40:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 418741248 bytes 2024-09-13T09:40:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 460466176 bytes 2024-09-13T09:40:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 463959040 bytes 2024-09-13T09:40:13.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 511269888 bytes 2024-09-13T09:40:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 543194112 bytes 2024-09-13T09:40:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 580287488 bytes 2024-09-13T09:40:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 627943424 bytes 2024-09-13T09:40:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 656139264 bytes 2024-09-13T09:40:14.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 710193152 bytes 2024-09-13T09:40:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 740948992 bytes 2024-09-13T09:40:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 773276672 bytes 2024-09-13T09:40:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 793503744 bytes 2024-09-13T09:40:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 823133184 bytes 2024-09-13T09:40:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 869646336 bytes 2024-09-13T09:40:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 936216576 bytes 2024-09-13T09:40:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 971436032 bytes 2024-09-13T09:40:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1022843904 bytes 2024-09-13T09:40:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1047599104 bytes 2024-09-13T09:40:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1067968512 bytes 2024-09-13T09:40:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1122743296 bytes 2024-09-13T09:40:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1198141440 bytes 2024-09-13T09:40:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1230597120 bytes 2024-09-13T09:40:16.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1247780864 bytes 2024-09-13T09:40:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1286323200 bytes 2024-09-13T09:40:16.511 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-13T09:40:16.511 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:40:16.511 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-13T09:40:16.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-13T09:40:16.512 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-13T09:40:16.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-13T09:40:16.516 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2024-09-13T09:40:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1334221824 bytes 2024-09-13T09:40:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1342213120 bytes 2024-09-13T09:40:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1367228416 bytes 2024-09-13T09:40:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1397777408 bytes 2024-09-13T09:40:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1435983872 bytes 2024-09-13T09:40:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1502508032 bytes 2024-09-13T09:40:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1540446208 bytes 2024-09-13T09:40:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1633760256 bytes 2024-09-13T09:40:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1710236672 bytes 2024-09-13T09:40:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1818037248 bytes 2024-09-13T09:40:17.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1867141120 bytes 2024-09-13T09:40:18.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1916873728 bytes 2024-09-13T09:40:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1917505536 bytes 2024-09-13T09:40:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1922652160 bytes 2024-09-13T09:40:18.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1923247104 bytes 2024-09-13T09:40:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1941835776 bytes 2024-09-13T09:40:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 1970217984 bytes 2024-09-13T09:40:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2008308736 bytes 2024-09-13T09:40:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2014863360 bytes 2024-09-13T09:40:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2065081344 bytes 2024-09-13T09:40:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2130209792 bytes 2024-09-13T09:40:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:40:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:40:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:40:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:40:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:40:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_26 2024-09-13T09:40:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2024-09-13T09:40:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:40:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:40:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:40:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_26 128M 2024-09-13T09:40:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:40:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:40:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:40:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:40:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_26 2024-09-13T09:40:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:40:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2024-09-13T09:40:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:40:19.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:40:19.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2024-09-13T09:40:20.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:40:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2024-09-13T09:40:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_26': 2024-09-13T09:40:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:40:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:40:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:40:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 137caa6eb12b 2024-09-13T09:40:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.137caa6eb12b 2024-09-13T09:40:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:40:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:40:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:40:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:40:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:40:19 2024 2024-09-13T09:40:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:40:19 2024 2024-09-13T09:40:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:40:19 2024 2024-09-13T09:40:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:40:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:40:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 9c358941-453b-4f18-9eae-0b0fdf040056 2024-09-13T09:40:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:40:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:40:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_26 2024-09-13T09:40:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:40:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:40:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2024-09-13T09:40:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:40:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:40:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 12101632 bytes 2024-09-13T09:40:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 85904384 bytes 2024-09-13T09:40:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 247235584 bytes 2024-09-13T09:40:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 287693824 bytes 2024-09-13T09:40:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 389775360 bytes 2024-09-13T09:40:21.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 471657472 bytes 2024-09-13T09:40:21.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 485737472 bytes 2024-09-13T09:40:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 524094464 bytes 2024-09-13T09:40:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 563444736 bytes 2024-09-13T09:40:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 586276864 bytes 2024-09-13T09:40:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 611492864 bytes 2024-09-13T09:40:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 642369536 bytes 2024-09-13T09:40:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 656169984 bytes 2024-09-13T09:40:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 698563584 bytes 2024-09-13T09:40:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 734603264 bytes 2024-09-13T09:40:22.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 826654720 bytes 2024-09-13T09:40:22.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 842315776 bytes 2024-09-13T09:40:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 848549888 bytes 2024-09-13T09:40:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 881874944 bytes 2024-09-13T09:40:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 916887552 bytes 2024-09-13T09:40:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 976365568 bytes 2024-09-13T09:40:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1024281600 bytes 2024-09-13T09:40:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1052630016 bytes 2024-09-13T09:40:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1106793472 bytes 2024-09-13T09:40:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1170177024 bytes 2024-09-13T09:40:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1208227840 bytes 2024-09-13T09:40:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1275642880 bytes 2024-09-13T09:40:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1292075008 bytes 2024-09-13T09:40:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1317473280 bytes 2024-09-13T09:40:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1352536064 bytes 2024-09-13T09:40:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1382309888 bytes 2024-09-13T09:40:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1399520256 bytes 2024-09-13T09:40:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1402725376 bytes 2024-09-13T09:40:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1450003456 bytes 2024-09-13T09:40:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1492282368 bytes 2024-09-13T09:40:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1556286464 bytes 2024-09-13T09:40:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1584245760 bytes 2024-09-13T09:40:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1654976512 bytes 2024-09-13T09:40:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1724054528 bytes 2024-09-13T09:40:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1797652480 bytes 2024-09-13T09:40:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1811512320 bytes 2024-09-13T09:40:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1834565632 bytes 2024-09-13T09:40:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1835167744 bytes 2024-09-13T09:40:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1972663296 bytes 2024-09-13T09:40:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2003454976 bytes 2024-09-13T09:40:26.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2085360640 bytes 2024-09-13T09:40:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2085978112 bytes 2024-09-13T09:40:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2115318784 bytes 2024-09-13T09:40:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2191112192 bytes 2024-09-13T09:40:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2209763328 bytes 2024-09-13T09:40:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2248768512 bytes 2024-09-13T09:40:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2251126784 bytes 2024-09-13T09:40:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2320421888 bytes 2024-09-13T09:40:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2366490624 bytes 2024-09-13T09:40:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2503400448 bytes 2024-09-13T09:40:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2529314816 bytes 2024-09-13T09:40:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2625470464 bytes 2024-09-13T09:40:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2679421952 bytes 2024-09-13T09:40:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2743268352 bytes 2024-09-13T09:40:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2788622336 bytes 2024-09-13T09:40:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2801500160 bytes 2024-09-13T09:40:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2853613568 bytes 2024-09-13T09:40:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2897429504 bytes 2024-09-13T09:40:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2916919296 bytes 2024-09-13T09:40:28.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2945392640 bytes 2024-09-13T09:40:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2978466816 bytes 2024-09-13T09:40:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 3020016640 bytes 2024-09-13T09:40:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 3102369792 bytes 2024-09-13T09:40:29.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 3131414528 bytes 2024-09-13T09:40:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 3189016576 bytes 2024-09-13T09:40:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 3216030720 bytes 2024-09-13T09:40:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 3253432320 bytes 2024-09-13T09:40:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 3279045632 bytes 2024-09-13T09:40:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3295154176 bytes 2024-09-13T09:40:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3295754240 bytes 2024-09-13T09:40:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3351911424 bytes 2024-09-13T09:40:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3461670912 bytes 2024-09-13T09:40:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3543662592 bytes 2024-09-13T09:40:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3559241728 bytes 2024-09-13T09:40:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3643341824 bytes 2024-09-13T09:40:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3701032960 bytes 2024-09-13T09:40:31.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3748436992 bytes 2024-09-13T09:40:31.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3792368640 bytes 2024-09-13T09:40:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3881451520 bytes 2024-09-13T09:40:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3903605760 bytes 2024-09-13T09:40:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3938190336 bytes 2024-09-13T09:40:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3978290176 bytes 2024-09-13T09:40:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 4005512192 bytes 2024-09-13T09:40:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 4008147968 bytes 2024-09-13T09:40:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:40:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:40:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:40:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:40:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:40:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_27 2024-09-13T09:40:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2024-09-13T09:40:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:40:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:40:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:40:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_27 128M 2024-09-13T09:40:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:40:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:40:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:40:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:40:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_27 2024-09-13T09:40:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:40:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2024-09-13T09:40:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:40:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:40:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2024-09-13T09:40:32.396 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-13T09:40:32.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-13T09:40:32.397 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-13T09:40:32.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:40:32.396+0000 7fa5b8253640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 43653) UID: 1000 2024-09-13T09:40:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:40:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2024-09-13T09:40:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_27': 2024-09-13T09:40:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:40:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:40:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:40:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 138680ffb486 2024-09-13T09:40:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.138680ffb486 2024-09-13T09:40:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:40:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:40:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:40:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:40:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:40:32 2024 2024-09-13T09:40:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:40:32 2024 2024-09-13T09:40:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:40:32 2024 2024-09-13T09:40:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:40:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:40:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 5ad6932e-f05d-4de5-89fb-186e6b5e26b1 2024-09-13T09:40:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:40:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:40:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_27 2024-09-13T09:40:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:40:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:40:32.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=8 2024-09-13T09:40:32.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:40:32.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:40:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 21814272 bytes 2024-09-13T09:40:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 61305856 bytes 2024-09-13T09:40:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 130683904 bytes 2024-09-13T09:40:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 233940992 bytes 2024-09-13T09:40:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 267583488 bytes 2024-09-13T09:40:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 294355968 bytes 2024-09-13T09:40:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 324951040 bytes 2024-09-13T09:40:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 330736640 bytes 2024-09-13T09:40:33.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 406128640 bytes 2024-09-13T09:40:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 478427136 bytes 2024-09-13T09:40:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 511258624 bytes 2024-09-13T09:40:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 522914816 bytes 2024-09-13T09:40:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 612316160 bytes 2024-09-13T09:40:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 612927488 bytes 2024-09-13T09:40:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 643139584 bytes 2024-09-13T09:40:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 691908608 bytes 2024-09-13T09:40:34.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 776672256 bytes 2024-09-13T09:40:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 828344320 bytes 2024-09-13T09:40:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 850747392 bytes 2024-09-13T09:40:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 888581120 bytes 2024-09-13T09:40:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 944441344 bytes 2024-09-13T09:40:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 962984960 bytes 2024-09-13T09:40:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1012291584 bytes 2024-09-13T09:40:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1054051328 bytes 2024-09-13T09:40:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1107374080 bytes 2024-09-13T09:40:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1254523904 bytes 2024-09-13T09:40:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1278375936 bytes 2024-09-13T09:40:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1304318976 bytes 2024-09-13T09:40:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1353985024 bytes 2024-09-13T09:40:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1383991296 bytes 2024-09-13T09:40:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1439924224 bytes 2024-09-13T09:40:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1476293632 bytes 2024-09-13T09:40:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1507152896 bytes 2024-09-13T09:40:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1530244096 bytes 2024-09-13T09:40:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1567387648 bytes 2024-09-13T09:40:37.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1572298752 bytes 2024-09-13T09:40:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1624068096 bytes 2024-09-13T09:40:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1699318784 bytes 2024-09-13T09:40:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1721728000 bytes 2024-09-13T09:40:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1740169216 bytes 2024-09-13T09:40:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1806857216 bytes 2024-09-13T09:40:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1858343936 bytes 2024-09-13T09:40:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1874417664 bytes 2024-09-13T09:40:38.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1986852864 bytes 2024-09-13T09:40:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1992511488 bytes 2024-09-13T09:40:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2011782144 bytes 2024-09-13T09:40:38.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2064812032 bytes 2024-09-13T09:40:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2156712960 bytes 2024-09-13T09:40:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2171679744 bytes 2024-09-13T09:40:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2236145664 bytes 2024-09-13T09:40:39.397 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-13T09:40:39.397 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:40:39.398 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-13T09:40:39.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-13T09:40:39.398 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-13T09:40:39.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-13T09:40:39.401 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2024-09-13T09:40:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2283218944 bytes 2024-09-13T09:40:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2353014784 bytes 2024-09-13T09:40:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2397010944 bytes 2024-09-13T09:40:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2441916416 bytes 2024-09-13T09:40:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2466116608 bytes 2024-09-13T09:40:40.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2550127616 bytes 2024-09-13T09:40:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2612532224 bytes 2024-09-13T09:40:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2680016896 bytes 2024-09-13T09:40:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:40:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:40:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:40:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:40:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:40:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_28 2024-09-13T09:40:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2024-09-13T09:40:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:40:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:40:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:40:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_28 128M 2024-09-13T09:40:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:40:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:40:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:40:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:40:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_28 2024-09-13T09:40:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:40:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2024-09-13T09:40:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:40:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:40:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2024-09-13T09:40:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:40:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2024-09-13T09:40:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_28': 2024-09-13T09:40:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:40:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:40:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:40:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1398467e1494 2024-09-13T09:40:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1398467e1494 2024-09-13T09:40:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:40:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:40:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:40:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:40:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:40:40 2024 2024-09-13T09:40:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:40:40 2024 2024-09-13T09:40:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:40:40 2024 2024-09-13T09:40:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:40:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:40:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: f0cfbdbd-fc4d-4303-a84d-1580b7778f89 2024-09-13T09:40:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:40:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:40:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_28 2024-09-13T09:40:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:40:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:40:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=5 2024-09-13T09:40:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:40:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:40:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 46620672 bytes 2024-09-13T09:40:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 65590272 bytes 2024-09-13T09:40:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 156935168 bytes 2024-09-13T09:40:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 309892096 bytes 2024-09-13T09:40:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 339217408 bytes 2024-09-13T09:40:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 395847680 bytes 2024-09-13T09:40:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 415810560 bytes 2024-09-13T09:40:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 416431104 bytes 2024-09-13T09:40:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 428598272 bytes 2024-09-13T09:40:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 447859712 bytes 2024-09-13T09:40:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 477424640 bytes 2024-09-13T09:40:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 517094400 bytes 2024-09-13T09:40:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 598487040 bytes 2024-09-13T09:40:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 626267136 bytes 2024-09-13T09:40:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 666358784 bytes 2024-09-13T09:40:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 747497472 bytes 2024-09-13T09:40:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 796425216 bytes 2024-09-13T09:40:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 808961024 bytes 2024-09-13T09:40:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 851019776 bytes 2024-09-13T09:40:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 895078400 bytes 2024-09-13T09:40:43.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 909403136 bytes 2024-09-13T09:40:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 966338560 bytes 2024-09-13T09:40:44.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1018764288 bytes 2024-09-13T09:40:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1078437888 bytes 2024-09-13T09:40:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1106791424 bytes 2024-09-13T09:40:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1176608768 bytes 2024-09-13T09:40:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1229176832 bytes 2024-09-13T09:40:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1254957056 bytes 2024-09-13T09:40:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1300120576 bytes 2024-09-13T09:40:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1344220160 bytes 2024-09-13T09:40:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1420672000 bytes 2024-09-13T09:40:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1451195392 bytes 2024-09-13T09:40:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1505780736 bytes 2024-09-13T09:40:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1550391296 bytes 2024-09-13T09:40:45.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1559427072 bytes 2024-09-13T09:40:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1591708672 bytes 2024-09-13T09:40:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1608239104 bytes 2024-09-13T09:40:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1654265856 bytes 2024-09-13T09:40:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:40:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:40:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:40:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:40:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:40:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_29 2024-09-13T09:40:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2024-09-13T09:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_29 128M 2024-09-13T09:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:40:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:40:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:40:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:40:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:40:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_29 2024-09-13T09:40:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:40:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2024-09-13T09:40:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:40:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:40:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2024-09-13T09:40:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:40:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2024-09-13T09:40:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_29': 2024-09-13T09:40:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:40:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:40:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:40:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13a05263c4f7 2024-09-13T09:40:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13a05263c4f7 2024-09-13T09:40:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:40:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:40:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:40:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:40:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:40:46 2024 2024-09-13T09:40:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:40:46 2024 2024-09-13T09:40:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:40:46 2024 2024-09-13T09:40:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:40:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:40:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: fea3fd66-9c9a-4468-9fc6-b7638eeb26b7 2024-09-13T09:40:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:40:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:40:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_29 2024-09-13T09:40:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:40:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:40:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:40:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:40:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=14 2024-09-13T09:40:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:40:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:40:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 42634240 bytes 2024-09-13T09:40:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 97017856 bytes 2024-09-13T09:40:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 128667648 bytes 2024-09-13T09:40:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 189985792 bytes 2024-09-13T09:40:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 214174720 bytes 2024-09-13T09:40:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 214784000 bytes 2024-09-13T09:40:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 272948224 bytes 2024-09-13T09:40:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 302503936 bytes 2024-09-13T09:40:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 324893696 bytes 2024-09-13T09:40:48.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 332823552 bytes 2024-09-13T09:40:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 373288960 bytes 2024-09-13T09:40:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 443361280 bytes 2024-09-13T09:40:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 509788160 bytes 2024-09-13T09:40:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 510658560 bytes 2024-09-13T09:40:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 523374592 bytes 2024-09-13T09:40:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 552511488 bytes 2024-09-13T09:40:49.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 600544256 bytes 2024-09-13T09:40:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 601138176 bytes 2024-09-13T09:40:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 639666176 bytes 2024-09-13T09:40:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 640702464 bytes 2024-09-13T09:40:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 700262400 bytes 2024-09-13T09:40:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 702277632 bytes 2024-09-13T09:40:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 775353344 bytes 2024-09-13T09:40:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 802055168 bytes 2024-09-13T09:40:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 850831360 bytes 2024-09-13T09:40:50.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 881138688 bytes 2024-09-13T09:40:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 957741056 bytes 2024-09-13T09:40:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 976886784 bytes 2024-09-13T09:40:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1039534080 bytes 2024-09-13T09:40:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1117737984 bytes 2024-09-13T09:40:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1169486848 bytes 2024-09-13T09:40:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1223874560 bytes 2024-09-13T09:40:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1240628224 bytes 2024-09-13T09:40:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1310684160 bytes 2024-09-13T09:40:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1386101760 bytes 2024-09-13T09:40:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1432712192 bytes 2024-09-13T09:40:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1472399360 bytes 2024-09-13T09:40:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1501721600 bytes 2024-09-13T09:40:52.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1593284608 bytes 2024-09-13T09:40:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1623668736 bytes 2024-09-13T09:40:52.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1691162624 bytes 2024-09-13T09:40:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1754785792 bytes 2024-09-13T09:40:52.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1876356096 bytes 2024-09-13T09:40:53.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1955760128 bytes 2024-09-13T09:40:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2001833984 bytes 2024-09-13T09:40:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2025125888 bytes 2024-09-13T09:40:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2025749504 bytes 2024-09-13T09:40:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2064929792 bytes 2024-09-13T09:40:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2116546560 bytes 2024-09-13T09:40:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2205388800 bytes 2024-09-13T09:40:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2228338688 bytes 2024-09-13T09:40:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2250716160 bytes 2024-09-13T09:40:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2291376128 bytes 2024-09-13T09:40:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2354586624 bytes 2024-09-13T09:40:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2372486144 bytes 2024-09-13T09:40:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2448803840 bytes 2024-09-13T09:40:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2506718208 bytes 2024-09-13T09:40:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2612588544 bytes 2024-09-13T09:40:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2666789888 bytes 2024-09-13T09:40:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2694814720 bytes 2024-09-13T09:40:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2752263168 bytes 2024-09-13T09:40:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2785537024 bytes 2024-09-13T09:40:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2814453760 bytes 2024-09-13T09:40:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2834628608 bytes 2024-09-13T09:40:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2935705600 bytes 2024-09-13T09:40:55.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2985558016 bytes 2024-09-13T09:40:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 3032553472 bytes 2024-09-13T09:40:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 3053967360 bytes 2024-09-13T09:40:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 3123061760 bytes 2024-09-13T09:40:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 3182453760 bytes 2024-09-13T09:40:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 3255938048 bytes 2024-09-13T09:40:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 3355699200 bytes 2024-09-13T09:40:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 3357030400 bytes 2024-09-13T09:40:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3493542912 bytes 2024-09-13T09:40:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3528809472 bytes 2024-09-13T09:40:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3553114112 bytes 2024-09-13T09:40:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3559941120 bytes 2024-09-13T09:40:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3640109056 bytes 2024-09-13T09:40:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3669071872 bytes 2024-09-13T09:40:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3737056256 bytes 2024-09-13T09:40:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3803394048 bytes 2024-09-13T09:40:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3879949312 bytes 2024-09-13T09:40:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3907867648 bytes 2024-09-13T09:40:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3955989504 bytes 2024-09-13T09:40:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3993816064 bytes 2024-09-13T09:40:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 4063560704 bytes 2024-09-13T09:40:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 4117956608 bytes 2024-09-13T09:40:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 4130135040 bytes 2024-09-13T09:40:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 4145688576 bytes 2024-09-13T09:40:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 4146304000 bytes 2024-09-13T09:40:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 4238555136 bytes 2024-09-13T09:40:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 4259960832 bytes 2024-09-13T09:40:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 4292336640 bytes 2024-09-13T09:40:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9400 IOs, 4294095872 bytes 2024-09-13T09:40:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9500 IOs, 4311577600 bytes 2024-09-13T09:40:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9600 IOs, 4381983744 bytes 2024-09-13T09:41:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9700 IOs, 4431025152 bytes 2024-09-13T09:41:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9800 IOs, 4454914048 bytes 2024-09-13T09:41:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9900 IOs, 4506258432 bytes 2024-09-13T09:41:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10000 IOs, 4520088576 bytes 2024-09-13T09:41:00.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10100 IOs, 4536314880 bytes 2024-09-13T09:41:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10200 IOs, 4601345024 bytes 2024-09-13T09:41:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10300 IOs, 4687512576 bytes 2024-09-13T09:41:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10400 IOs, 4782023680 bytes 2024-09-13T09:41:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:41:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:41:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:41:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:41:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:41:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_30 2024-09-13T09:41:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2024-09-13T09:41:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:41:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:41:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:41:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_30 128M 2024-09-13T09:41:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:41:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:41:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:41:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:41:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_30 2024-09-13T09:41:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:41:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2024-09-13T09:41:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:41:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:41:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2024-09-13T09:41:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:41:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2024-09-13T09:41:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_30': 2024-09-13T09:41:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:41:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:41:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:41:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13a8ff4a1fc8 2024-09-13T09:41:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13a8ff4a1fc8 2024-09-13T09:41:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:41:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:41:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:41:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:41:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:41:01 2024 2024-09-13T09:41:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:41:01 2024 2024-09-13T09:41:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:41:01 2024 2024-09-13T09:41:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:41:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:41:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 1b942165-b9f5-4c05-8666-d8d9900c56b6 2024-09-13T09:41:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:41:02.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:41:02.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_30 2024-09-13T09:41:02.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:02.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:02.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:41:02.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:41:02.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=13 2024-09-13T09:41:02.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:41:02.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:41:02.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 609280 bytes 2024-09-13T09:41:02.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 18820096 bytes 2024-09-13T09:41:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 30016512 bytes 2024-09-13T09:41:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 30642176 bytes 2024-09-13T09:41:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 80995328 bytes 2024-09-13T09:41:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 181826560 bytes 2024-09-13T09:41:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 207668224 bytes 2024-09-13T09:41:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 246167552 bytes 2024-09-13T09:41:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 246770688 bytes 2024-09-13T09:41:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 255423488 bytes 2024-09-13T09:41:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 368025600 bytes 2024-09-13T09:41:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 379200512 bytes 2024-09-13T09:41:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 418407424 bytes 2024-09-13T09:41:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 436604928 bytes 2024-09-13T09:41:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 525869056 bytes 2024-09-13T09:41:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 537801728 bytes 2024-09-13T09:41:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 550468608 bytes 2024-09-13T09:41:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 621262848 bytes 2024-09-13T09:41:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 671739904 bytes 2024-09-13T09:41:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 760540160 bytes 2024-09-13T09:41:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 817013760 bytes 2024-09-13T09:41:05.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 837903360 bytes 2024-09-13T09:41:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 860012544 bytes 2024-09-13T09:41:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 871459840 bytes 2024-09-13T09:41:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 955381760 bytes 2024-09-13T09:41:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1002577920 bytes 2024-09-13T09:41:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1053194240 bytes 2024-09-13T09:41:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1057036288 bytes 2024-09-13T09:41:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1071019008 bytes 2024-09-13T09:41:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1093112832 bytes 2024-09-13T09:41:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1137467392 bytes 2024-09-13T09:41:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1179334656 bytes 2024-09-13T09:41:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1232741376 bytes 2024-09-13T09:41:06.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1260398592 bytes 2024-09-13T09:41:06.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1271664640 bytes 2024-09-13T09:41:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1314843648 bytes 2024-09-13T09:41:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1381910528 bytes 2024-09-13T09:41:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1404877824 bytes 2024-09-13T09:41:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1405520896 bytes 2024-09-13T09:41:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1415956480 bytes 2024-09-13T09:41:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1455237120 bytes 2024-09-13T09:41:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1505128448 bytes 2024-09-13T09:41:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1518187520 bytes 2024-09-13T09:41:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1576366080 bytes 2024-09-13T09:41:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1647232000 bytes 2024-09-13T09:41:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1719917568 bytes 2024-09-13T09:41:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1752434688 bytes 2024-09-13T09:41:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1769470976 bytes 2024-09-13T09:41:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1821721600 bytes 2024-09-13T09:41:08.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1902297088 bytes 2024-09-13T09:41:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 1929363456 bytes 2024-09-13T09:41:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 1941040128 bytes 2024-09-13T09:41:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 1987318784 bytes 2024-09-13T09:41:08.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 1987921920 bytes 2024-09-13T09:41:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2049976320 bytes 2024-09-13T09:41:09.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2073043968 bytes 2024-09-13T09:41:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2116920320 bytes 2024-09-13T09:41:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2223470592 bytes 2024-09-13T09:41:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2304513024 bytes 2024-09-13T09:41:09.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2350050304 bytes 2024-09-13T09:41:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2382679040 bytes 2024-09-13T09:41:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2408467456 bytes 2024-09-13T09:41:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2444214272 bytes 2024-09-13T09:41:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2536130560 bytes 2024-09-13T09:41:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2551652352 bytes 2024-09-13T09:41:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2597698560 bytes 2024-09-13T09:41:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2621565952 bytes 2024-09-13T09:41:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2684653568 bytes 2024-09-13T09:41:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2762776576 bytes 2024-09-13T09:41:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2826414080 bytes 2024-09-13T09:41:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2854695936 bytes 2024-09-13T09:41:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2855294976 bytes 2024-09-13T09:41:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2963534848 bytes 2024-09-13T09:41:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 2965652480 bytes 2024-09-13T09:41:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3008020480 bytes 2024-09-13T09:41:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3046882304 bytes 2024-09-13T09:41:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3094986752 bytes 2024-09-13T09:41:12.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3152794624 bytes 2024-09-13T09:41:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3190039552 bytes 2024-09-13T09:41:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3212242944 bytes 2024-09-13T09:41:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3255987200 bytes 2024-09-13T09:41:12.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3270006784 bytes 2024-09-13T09:41:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3278599168 bytes 2024-09-13T09:41:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3291816960 bytes 2024-09-13T09:41:12.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3333489664 bytes 2024-09-13T09:41:13.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3344261120 bytes 2024-09-13T09:41:13.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3352049664 bytes 2024-09-13T09:41:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3377992704 bytes 2024-09-13T09:41:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3389479936 bytes 2024-09-13T09:41:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 3462318080 bytes 2024-09-13T09:41:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 3505440768 bytes 2024-09-13T09:41:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 3550926848 bytes 2024-09-13T09:41:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 3566124032 bytes 2024-09-13T09:41:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9400 IOs, 3589454848 bytes 2024-09-13T09:41:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9500 IOs, 3629483008 bytes 2024-09-13T09:41:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9600 IOs, 3657555968 bytes 2024-09-13T09:41:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9700 IOs, 3734894592 bytes 2024-09-13T09:41:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9800 IOs, 3772862464 bytes 2024-09-13T09:41:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9900 IOs, 3797371904 bytes 2024-09-13T09:41:14.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10000 IOs, 3842249728 bytes 2024-09-13T09:41:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10100 IOs, 3856297984 bytes 2024-09-13T09:41:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10200 IOs, 3889456128 bytes 2024-09-13T09:41:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10300 IOs, 3910089728 bytes 2024-09-13T09:41:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10400 IOs, 3940178944 bytes 2024-09-13T09:41:15.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:41:15.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:41:15.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:41:15.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:41:15.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:41:15.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_31 2024-09-13T09:41:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2024-09-13T09:41:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:41:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:41:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:41:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_31 128M 2024-09-13T09:41:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:41:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:41:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:41:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:41:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_31 2024-09-13T09:41:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:41:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2024-09-13T09:41:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:41:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:41:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2024-09-13T09:41:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:41:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2024-09-13T09:41:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_31': 2024-09-13T09:41:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:41:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:41:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:41:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13b2d320caa7 2024-09-13T09:41:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13b2d320caa7 2024-09-13T09:41:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:41:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:41:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:41:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:41:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:41:15 2024 2024-09-13T09:41:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:41:15 2024 2024-09-13T09:41:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:41:15 2024 2024-09-13T09:41:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:41:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:41:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 1bb5e3e6-31d3-4cca-8cd7-40225052afae 2024-09-13T09:41:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:41:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:41:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_31 2024-09-13T09:41:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:41:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:41:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=7 2024-09-13T09:41:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:41:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:41:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 23612416 bytes 2024-09-13T09:41:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 24225792 bytes 2024-09-13T09:41:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 24841216 bytes 2024-09-13T09:41:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 25451520 bytes 2024-09-13T09:41:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 85538816 bytes 2024-09-13T09:41:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 108547072 bytes 2024-09-13T09:41:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 160068608 bytes 2024-09-13T09:41:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 194279424 bytes 2024-09-13T09:41:16.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 241487872 bytes 2024-09-13T09:41:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 281559040 bytes 2024-09-13T09:41:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 337000448 bytes 2024-09-13T09:41:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 337624064 bytes 2024-09-13T09:41:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 391061504 bytes 2024-09-13T09:41:17.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 449003520 bytes 2024-09-13T09:41:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 524634112 bytes 2024-09-13T09:41:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 581890048 bytes 2024-09-13T09:41:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 651003904 bytes 2024-09-13T09:41:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 729885696 bytes 2024-09-13T09:41:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 753013760 bytes 2024-09-13T09:41:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 840137728 bytes 2024-09-13T09:41:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 874978304 bytes 2024-09-13T09:41:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 984636416 bytes 2024-09-13T09:41:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1070637056 bytes 2024-09-13T09:41:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1092702208 bytes 2024-09-13T09:41:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1167395840 bytes 2024-09-13T09:41:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1168021504 bytes 2024-09-13T09:41:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1192790016 bytes 2024-09-13T09:41:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1224936448 bytes 2024-09-13T09:41:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1281039360 bytes 2024-09-13T09:41:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1294835712 bytes 2024-09-13T09:41:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1304581120 bytes 2024-09-13T09:41:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1350405120 bytes 2024-09-13T09:41:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1429657600 bytes 2024-09-13T09:41:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1528103936 bytes 2024-09-13T09:41:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1577573376 bytes 2024-09-13T09:41:20.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1602032640 bytes 2024-09-13T09:41:20.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1602625536 bytes 2024-09-13T09:41:20.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1678084096 bytes 2024-09-13T09:41:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1737463808 bytes 2024-09-13T09:41:20.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1787617280 bytes 2024-09-13T09:41:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1817476096 bytes 2024-09-13T09:41:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1852405760 bytes 2024-09-13T09:41:21.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1926593536 bytes 2024-09-13T09:41:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1953347584 bytes 2024-09-13T09:41:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1983971328 bytes 2024-09-13T09:41:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2070589440 bytes 2024-09-13T09:41:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2102225920 bytes 2024-09-13T09:41:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2105698304 bytes 2024-09-13T09:41:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2163264512 bytes 2024-09-13T09:41:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2191288320 bytes 2024-09-13T09:41:22.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2262662144 bytes 2024-09-13T09:41:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2326737920 bytes 2024-09-13T09:41:22.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:41:22.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:41:22.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:41:22.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:41:22.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:41:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_32 2024-09-13T09:41:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2024-09-13T09:41:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:41:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:41:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:41:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_32 128M 2024-09-13T09:41:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:41:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:41:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:41:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:41:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_32 2024-09-13T09:41:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:41:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2024-09-13T09:41:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:41:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:41:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2024-09-13T09:41:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:41:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2024-09-13T09:41:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_32': 2024-09-13T09:41:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:41:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:41:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:41:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13ba2a07e0ef 2024-09-13T09:41:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13ba2a07e0ef 2024-09-13T09:41:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:41:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:41:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:41:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:41:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:41:22 2024 2024-09-13T09:41:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:41:22 2024 2024-09-13T09:41:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:41:22 2024 2024-09-13T09:41:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:41:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:41:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: ae63b672-de66-4994-b9e2-9cc620b8e9ef 2024-09-13T09:41:23.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:41:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:41:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_32 2024-09-13T09:41:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:41:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:41:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=7 2024-09-13T09:41:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:41:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:41:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 51335168 bytes 2024-09-13T09:41:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 93359104 bytes 2024-09-13T09:41:24.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 160285696 bytes 2024-09-13T09:41:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 197867520 bytes 2024-09-13T09:41:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 251250688 bytes 2024-09-13T09:41:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 328259584 bytes 2024-09-13T09:41:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 441951232 bytes 2024-09-13T09:41:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 450651136 bytes 2024-09-13T09:41:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 490935296 bytes 2024-09-13T09:41:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 498352128 bytes 2024-09-13T09:41:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 499924992 bytes 2024-09-13T09:41:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 500539392 bytes 2024-09-13T09:41:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 558039040 bytes 2024-09-13T09:41:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 677108736 bytes 2024-09-13T09:41:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 717032448 bytes 2024-09-13T09:41:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 759535616 bytes 2024-09-13T09:41:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 764905472 bytes 2024-09-13T09:41:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 829450240 bytes 2024-09-13T09:41:26.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 841492480 bytes 2024-09-13T09:41:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 918919168 bytes 2024-09-13T09:41:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 955893760 bytes 2024-09-13T09:41:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 956509184 bytes 2024-09-13T09:41:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1032147968 bytes 2024-09-13T09:41:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1099736064 bytes 2024-09-13T09:41:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1100342272 bytes 2024-09-13T09:41:27.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1121534976 bytes 2024-09-13T09:41:27.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1163028480 bytes 2024-09-13T09:41:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1228591104 bytes 2024-09-13T09:41:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1300344832 bytes 2024-09-13T09:41:27.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1307326464 bytes 2024-09-13T09:41:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1352088576 bytes 2024-09-13T09:41:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1365122048 bytes 2024-09-13T09:41:28.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1365723136 bytes 2024-09-13T09:41:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1466090496 bytes 2024-09-13T09:41:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1478453248 bytes 2024-09-13T09:41:28.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1537651712 bytes 2024-09-13T09:41:28.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1562452992 bytes 2024-09-13T09:41:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1655561216 bytes 2024-09-13T09:41:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1691229184 bytes 2024-09-13T09:41:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1706946560 bytes 2024-09-13T09:41:29.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1749964800 bytes 2024-09-13T09:41:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1784624128 bytes 2024-09-13T09:41:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1833689088 bytes 2024-09-13T09:41:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1931106304 bytes 2024-09-13T09:41:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1975459840 bytes 2024-09-13T09:41:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2040877056 bytes 2024-09-13T09:41:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2083306496 bytes 2024-09-13T09:41:30.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2138372096 bytes 2024-09-13T09:41:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2177380352 bytes 2024-09-13T09:41:30.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2234229760 bytes 2024-09-13T09:41:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2311975936 bytes 2024-09-13T09:41:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:41:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:41:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:41:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:41:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:41:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_33 2024-09-13T09:41:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2024-09-13T09:41:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:41:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:41:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:41:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_33 128M 2024-09-13T09:41:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:41:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:41:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:41:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:41:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_33 2024-09-13T09:41:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:41:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2024-09-13T09:41:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:41:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:41:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2024-09-13T09:41:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:41:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2024-09-13T09:41:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_33': 2024-09-13T09:41:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:41:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:41:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:41:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13c363ad7386 2024-09-13T09:41:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13c363ad7386 2024-09-13T09:41:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:41:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:41:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:41:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:41:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:41:30 2024 2024-09-13T09:41:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:41:30 2024 2024-09-13T09:41:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:41:30 2024 2024-09-13T09:41:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:41:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:41:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 7219b237-53fc-4501-baae-6c41f736571a 2024-09-13T09:41:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:41:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:41:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_33 2024-09-13T09:41:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:41:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:41:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=10 2024-09-13T09:41:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:41:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:41:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 34391040 bytes 2024-09-13T09:41:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 98622464 bytes 2024-09-13T09:41:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 99274752 bytes 2024-09-13T09:41:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 136696832 bytes 2024-09-13T09:41:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 189479936 bytes 2024-09-13T09:41:32.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 223210496 bytes 2024-09-13T09:41:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 227857408 bytes 2024-09-13T09:41:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 270055424 bytes 2024-09-13T09:41:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 294275072 bytes 2024-09-13T09:41:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 377721856 bytes 2024-09-13T09:41:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 409380864 bytes 2024-09-13T09:41:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 462814208 bytes 2024-09-13T09:41:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 546508800 bytes 2024-09-13T09:41:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 551536640 bytes 2024-09-13T09:41:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 565786624 bytes 2024-09-13T09:41:33.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 622735360 bytes 2024-09-13T09:41:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 642611200 bytes 2024-09-13T09:41:34.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 666196992 bytes 2024-09-13T09:41:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 695506944 bytes 2024-09-13T09:41:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 744584192 bytes 2024-09-13T09:41:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 862320640 bytes 2024-09-13T09:41:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 937085952 bytes 2024-09-13T09:41:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 987240448 bytes 2024-09-13T09:41:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1015639040 bytes 2024-09-13T09:41:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1039993856 bytes 2024-09-13T09:41:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1086417920 bytes 2024-09-13T09:41:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1147019264 bytes 2024-09-13T09:41:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1170075648 bytes 2024-09-13T09:41:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1184619520 bytes 2024-09-13T09:41:35.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1230369792 bytes 2024-09-13T09:41:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1231248384 bytes 2024-09-13T09:41:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1253010432 bytes 2024-09-13T09:41:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1302762496 bytes 2024-09-13T09:41:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1350700032 bytes 2024-09-13T09:41:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1362605056 bytes 2024-09-13T09:41:36.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1425471488 bytes 2024-09-13T09:41:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1485611008 bytes 2024-09-13T09:41:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1544649728 bytes 2024-09-13T09:41:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1628928000 bytes 2024-09-13T09:41:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1661274112 bytes 2024-09-13T09:41:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1725931520 bytes 2024-09-13T09:41:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1767243776 bytes 2024-09-13T09:41:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1809381376 bytes 2024-09-13T09:41:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1833221120 bytes 2024-09-13T09:41:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1879754752 bytes 2024-09-13T09:41:37.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1921255424 bytes 2024-09-13T09:41:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1949623296 bytes 2024-09-13T09:41:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1957028864 bytes 2024-09-13T09:41:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1978698752 bytes 2024-09-13T09:41:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2026797056 bytes 2024-09-13T09:41:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2028217344 bytes 2024-09-13T09:41:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2073200640 bytes 2024-09-13T09:41:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2142578688 bytes 2024-09-13T09:41:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2155415552 bytes 2024-09-13T09:41:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2200600576 bytes 2024-09-13T09:41:38.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2216170496 bytes 2024-09-13T09:41:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2253673472 bytes 2024-09-13T09:41:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2273064960 bytes 2024-09-13T09:41:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2326177792 bytes 2024-09-13T09:41:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2418696192 bytes 2024-09-13T09:41:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2474954752 bytes 2024-09-13T09:41:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2492921856 bytes 2024-09-13T09:41:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2532017152 bytes 2024-09-13T09:41:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2532641792 bytes 2024-09-13T09:41:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2582900736 bytes 2024-09-13T09:41:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2608353280 bytes 2024-09-13T09:41:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2706226176 bytes 2024-09-13T09:41:40.401 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3740967876931254) > weight (0.25) 2024-09-13T09:41:40.401 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8597278010021445) > weight (0.25) 2024-09-13T09:41:40.401 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-13T09:41:40.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-13T09:41:40.402 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-13T09:41:40.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:41:40.401+0000 7fcf38177640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 39051) UID: 1000 2024-09-13T09:41:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2706853888 bytes 2024-09-13T09:41:40.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2845329408 bytes 2024-09-13T09:41:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2853594112 bytes 2024-09-13T09:41:40.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2882501632 bytes 2024-09-13T09:41:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2949304320 bytes 2024-09-13T09:41:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2949895168 bytes 2024-09-13T09:41:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 2950507520 bytes 2024-09-13T09:41:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3034707968 bytes 2024-09-13T09:41:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3088480256 bytes 2024-09-13T09:41:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:41:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:41:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:41:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:41:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:41:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_34 2024-09-13T09:41:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2024-09-13T09:41:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:41:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:41:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:41:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_34 128M 2024-09-13T09:41:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:41:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:41:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:41:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:41:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_34 2024-09-13T09:41:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:41:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2024-09-13T09:41:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:41:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:41:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2024-09-13T09:41:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:41:42.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2024-09-13T09:41:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_34': 2024-09-13T09:41:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:41:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13cc58876d1 2024-09-13T09:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13cc58876d1 2024-09-13T09:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:41:41 2024 2024-09-13T09:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:41:41 2024 2024-09-13T09:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:41:41 2024 2024-09-13T09:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 7902759b-4c55-479e-b92d-2527b62dd9df 2024-09-13T09:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:41:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:41:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_34 2024-09-13T09:41:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:41:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:41:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=5 2024-09-13T09:41:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:41:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:41:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 598016 bytes 2024-09-13T09:41:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 85513216 bytes 2024-09-13T09:41:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 159058944 bytes 2024-09-13T09:41:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 192593920 bytes 2024-09-13T09:41:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 202646528 bytes 2024-09-13T09:41:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 214224896 bytes 2024-09-13T09:41:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 267298816 bytes 2024-09-13T09:41:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 287335424 bytes 2024-09-13T09:41:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 325956608 bytes 2024-09-13T09:41:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 346153984 bytes 2024-09-13T09:41:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 380891136 bytes 2024-09-13T09:41:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 471301120 bytes 2024-09-13T09:41:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 477798400 bytes 2024-09-13T09:41:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 497227776 bytes 2024-09-13T09:41:44.403 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-13T09:41:44.403 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:41:44.403 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-13T09:41:44.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-13T09:41:44.404 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-13T09:41:44.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-13T09:41:44.407 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2024-09-13T09:41:44.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 559522816 bytes 2024-09-13T09:41:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 592526336 bytes 2024-09-13T09:41:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 609086464 bytes 2024-09-13T09:41:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 657569792 bytes 2024-09-13T09:41:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 699355136 bytes 2024-09-13T09:41:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 783206400 bytes 2024-09-13T09:41:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 806568960 bytes 2024-09-13T09:41:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 810496000 bytes 2024-09-13T09:41:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 870231040 bytes 2024-09-13T09:41:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 932159488 bytes 2024-09-13T09:41:46.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1051527168 bytes 2024-09-13T09:41:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1064838144 bytes 2024-09-13T09:41:46.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1121590272 bytes 2024-09-13T09:41:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1134225408 bytes 2024-09-13T09:41:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1235529728 bytes 2024-09-13T09:41:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1335400448 bytes 2024-09-13T09:41:46.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1399880704 bytes 2024-09-13T09:41:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1415501824 bytes 2024-09-13T09:41:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1512015872 bytes 2024-09-13T09:41:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1560004608 bytes 2024-09-13T09:41:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1603977216 bytes 2024-09-13T09:41:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1683529728 bytes 2024-09-13T09:41:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:41:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:41:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:41:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:41:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:41:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_35 2024-09-13T09:41:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2024-09-13T09:41:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:41:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:41:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:41:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_35 128M 2024-09-13T09:41:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:41:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:41:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:41:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:41:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_35 2024-09-13T09:41:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:41:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2024-09-13T09:41:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:41:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:41:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2024-09-13T09:41:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:41:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2024-09-13T09:41:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_35': 2024-09-13T09:41:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:41:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:41:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:41:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13e1577ef9a4 2024-09-13T09:41:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13e1577ef9a4 2024-09-13T09:41:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:41:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:41:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:41:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:41:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:41:47 2024 2024-09-13T09:41:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:41:47 2024 2024-09-13T09:41:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:41:47 2024 2024-09-13T09:41:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:41:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:41:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 6cbe51d3-590c-415c-bec2-78c533fae3f5 2024-09-13T09:41:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:41:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:41:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_35 2024-09-13T09:41:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:41:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:41:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=6 2024-09-13T09:41:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:41:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:41:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 36398080 bytes 2024-09-13T09:41:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 79589376 bytes 2024-09-13T09:41:49.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 124148736 bytes 2024-09-13T09:41:49.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 187100160 bytes 2024-09-13T09:41:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 251761664 bytes 2024-09-13T09:41:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 252401664 bytes 2024-09-13T09:41:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 305539072 bytes 2024-09-13T09:41:49.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 388391936 bytes 2024-09-13T09:41:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 424871936 bytes 2024-09-13T09:41:50.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 514164736 bytes 2024-09-13T09:41:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 575295488 bytes 2024-09-13T09:41:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 575913984 bytes 2024-09-13T09:41:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 602945536 bytes 2024-09-13T09:41:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 648713216 bytes 2024-09-13T09:41:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 707237888 bytes 2024-09-13T09:41:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 745158656 bytes 2024-09-13T09:41:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 774007808 bytes 2024-09-13T09:41:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 825357312 bytes 2024-09-13T09:41:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 828914688 bytes 2024-09-13T09:41:51.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 894575616 bytes 2024-09-13T09:41:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 910353408 bytes 2024-09-13T09:41:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 950712320 bytes 2024-09-13T09:41:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 975673344 bytes 2024-09-13T09:41:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1043453952 bytes 2024-09-13T09:41:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1092945920 bytes 2024-09-13T09:41:51.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1093579776 bytes 2024-09-13T09:41:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1168530432 bytes 2024-09-13T09:41:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1223697408 bytes 2024-09-13T09:41:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1229046784 bytes 2024-09-13T09:41:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1327367168 bytes 2024-09-13T09:41:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1379177472 bytes 2024-09-13T09:41:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1389192192 bytes 2024-09-13T09:41:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1420822528 bytes 2024-09-13T09:41:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1444026368 bytes 2024-09-13T09:41:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1482888192 bytes 2024-09-13T09:41:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1529044992 bytes 2024-09-13T09:41:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1592252416 bytes 2024-09-13T09:41:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1647628288 bytes 2024-09-13T09:41:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1756675072 bytes 2024-09-13T09:41:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1804171264 bytes 2024-09-13T09:41:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1839423488 bytes 2024-09-13T09:41:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1893784576 bytes 2024-09-13T09:41:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1927703552 bytes 2024-09-13T09:41:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1974290432 bytes 2024-09-13T09:41:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2043008000 bytes 2024-09-13T09:41:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:41:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:41:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:41:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:41:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:41:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_36 2024-09-13T09:41:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2024-09-13T09:41:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:41:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:41:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:41:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_36 128M 2024-09-13T09:41:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:41:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:41:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:41:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:41:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:41:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_36 2024-09-13T09:41:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:41:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2024-09-13T09:41:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:41:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:41:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2024-09-13T09:41:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:41:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2024-09-13T09:41:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_36': 2024-09-13T09:41:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:41:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:41:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:41:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13ed9b764cc3 2024-09-13T09:41:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13ed9b764cc3 2024-09-13T09:41:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:41:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:41:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:41:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:41:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:41:54 2024 2024-09-13T09:41:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:41:54 2024 2024-09-13T09:41:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:41:54 2024 2024-09-13T09:41:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:41:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:41:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 9f803efc-9109-44ce-9b95-114b6dc86252 2024-09-13T09:41:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:41:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:41:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_36 2024-09-13T09:41:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:41:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:41:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:41:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:41:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2024-09-13T09:41:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:41:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:41:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 53551104 bytes 2024-09-13T09:41:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 70208512 bytes 2024-09-13T09:41:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 123064320 bytes 2024-09-13T09:41:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 225108992 bytes 2024-09-13T09:41:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 251080704 bytes 2024-09-13T09:41:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 297775104 bytes 2024-09-13T09:41:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 375313408 bytes 2024-09-13T09:41:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 455122944 bytes 2024-09-13T09:41:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 501150720 bytes 2024-09-13T09:41:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 607532032 bytes 2024-09-13T09:41:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 619267072 bytes 2024-09-13T09:41:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 667146240 bytes 2024-09-13T09:41:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 725934080 bytes 2024-09-13T09:41:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 791613440 bytes 2024-09-13T09:41:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 816209920 bytes 2024-09-13T09:41:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 914071552 bytes 2024-09-13T09:41:58.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 958379008 bytes 2024-09-13T09:41:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 1016112128 bytes 2024-09-13T09:41:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 1016740864 bytes 2024-09-13T09:41:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 1058700288 bytes 2024-09-13T09:41:58.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1070734336 bytes 2024-09-13T09:41:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1118925824 bytes 2024-09-13T09:41:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1137169408 bytes 2024-09-13T09:41:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1223884800 bytes 2024-09-13T09:41:59.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1314656256 bytes 2024-09-13T09:41:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1354809344 bytes 2024-09-13T09:41:59.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1424247808 bytes 2024-09-13T09:41:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1454163968 bytes 2024-09-13T09:41:59.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1514281984 bytes 2024-09-13T09:41:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1584284672 bytes 2024-09-13T09:42:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1632982016 bytes 2024-09-13T09:42:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1702865920 bytes 2024-09-13T09:42:00.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1770106880 bytes 2024-09-13T09:42:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1785745408 bytes 2024-09-13T09:42:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1838215168 bytes 2024-09-13T09:42:00.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1866836992 bytes 2024-09-13T09:42:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1878520832 bytes 2024-09-13T09:42:01.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1963374592 bytes 2024-09-13T09:42:01.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1999894528 bytes 2024-09-13T09:42:01.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 2058735616 bytes 2024-09-13T09:42:01.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 2117035008 bytes 2024-09-13T09:42:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 2224122880 bytes 2024-09-13T09:42:01.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 2241489920 bytes 2024-09-13T09:42:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2290539520 bytes 2024-09-13T09:42:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2295207936 bytes 2024-09-13T09:42:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2321278976 bytes 2024-09-13T09:42:02.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2328640512 bytes 2024-09-13T09:42:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2329246720 bytes 2024-09-13T09:42:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2329859072 bytes 2024-09-13T09:42:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2372120576 bytes 2024-09-13T09:42:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2407565312 bytes 2024-09-13T09:42:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2423815168 bytes 2024-09-13T09:42:03.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2503527424 bytes 2024-09-13T09:42:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2549171200 bytes 2024-09-13T09:42:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2611136512 bytes 2024-09-13T09:42:03.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2651581440 bytes 2024-09-13T09:42:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2729641984 bytes 2024-09-13T09:42:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2748929024 bytes 2024-09-13T09:42:04.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2898294784 bytes 2024-09-13T09:42:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2936528896 bytes 2024-09-13T09:42:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 3007632384 bytes 2024-09-13T09:42:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 3083460608 bytes 2024-09-13T09:42:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 3117419520 bytes 2024-09-13T09:42:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 3118016512 bytes 2024-09-13T09:42:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 3147990016 bytes 2024-09-13T09:42:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 3148605440 bytes 2024-09-13T09:42:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 3184530432 bytes 2024-09-13T09:42:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 3235837952 bytes 2024-09-13T09:42:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 3277375488 bytes 2024-09-13T09:42:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 3311609856 bytes 2024-09-13T09:42:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 3368384512 bytes 2024-09-13T09:42:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 3439320064 bytes 2024-09-13T09:42:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 3460816896 bytes 2024-09-13T09:42:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3487067136 bytes 2024-09-13T09:42:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3487724544 bytes 2024-09-13T09:42:06.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3497059328 bytes 2024-09-13T09:42:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3577000960 bytes 2024-09-13T09:42:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3629544448 bytes 2024-09-13T09:42:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3630159872 bytes 2024-09-13T09:42:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3672894464 bytes 2024-09-13T09:42:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3707334656 bytes 2024-09-13T09:42:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3733159936 bytes 2024-09-13T09:42:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3787340800 bytes 2024-09-13T09:42:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3815495680 bytes 2024-09-13T09:42:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3884333056 bytes 2024-09-13T09:42:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:42:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:42:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:42:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:42:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:42:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_37 2024-09-13T09:42:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2024-09-13T09:42:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:42:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:42:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:42:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:42:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:42:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_37 128M 2024-09-13T09:42:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:42:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:42:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:42:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:42:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:42:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:42:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_37 2024-09-13T09:42:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:42:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2024-09-13T09:42:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:42:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:42:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:42:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:42:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2024-09-13T09:42:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:42:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2024-09-13T09:42:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_37': 2024-09-13T09:42:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:42:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:42:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:42:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 13f9f3e48ab5 2024-09-13T09:42:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.13f9f3e48ab5 2024-09-13T09:42:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:42:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:42:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:42:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:42:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:42:07 2024 2024-09-13T09:42:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:42:07 2024 2024-09-13T09:42:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:42:07 2024 2024-09-13T09:42:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:42:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:42:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 2428288c-879f-45c0-a7c5-4f2760c5f863 2024-09-13T09:42:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:42:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:42:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_37 2024-09-13T09:42:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:42:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:42:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:42:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:42:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=9 2024-09-13T09:42:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:42:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:42:08.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 103581696 bytes 2024-09-13T09:42:09.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 188377088 bytes 2024-09-13T09:42:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 210497536 bytes 2024-09-13T09:42:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 253185024 bytes 2024-09-13T09:42:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 307372032 bytes 2024-09-13T09:42:09.517 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3143942925022354) > weight (0.25) 2024-09-13T09:42:09.517 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9835959250378479) > weight (0.25) 2024-09-13T09:42:09.517 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4694219718612255) > weight (0.25) 2024-09-13T09:42:09.517 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6526862474844839) > weight (0.25) 2024-09-13T09:42:09.517 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-09-13T09:42:09.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 307970048 bytes 2024-09-13T09:42:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 375272448 bytes 2024-09-13T09:42:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 386023424 bytes 2024-09-13T09:42:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 461026304 bytes 2024-09-13T09:42:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 494419968 bytes 2024-09-13T09:42:10.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 513481728 bytes 2024-09-13T09:42:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 531288064 bytes 2024-09-13T09:42:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 578603008 bytes 2024-09-13T09:42:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 640607232 bytes 2024-09-13T09:42:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 739544064 bytes 2024-09-13T09:42:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 788325376 bytes 2024-09-13T09:42:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 817669120 bytes 2024-09-13T09:42:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 868792320 bytes 2024-09-13T09:42:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 919110656 bytes 2024-09-13T09:42:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 968303616 bytes 2024-09-13T09:42:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1012316160 bytes 2024-09-13T09:42:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1035793408 bytes 2024-09-13T09:42:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1107016704 bytes 2024-09-13T09:42:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1135872000 bytes 2024-09-13T09:42:12.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1192736768 bytes 2024-09-13T09:42:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1215612928 bytes 2024-09-13T09:42:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1219001344 bytes 2024-09-13T09:42:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1232203776 bytes 2024-09-13T09:42:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1285654528 bytes 2024-09-13T09:42:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1314854912 bytes 2024-09-13T09:42:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1342852096 bytes 2024-09-13T09:42:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1376148480 bytes 2024-09-13T09:42:13.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1425696768 bytes 2024-09-13T09:42:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1463847936 bytes 2024-09-13T09:42:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1507189760 bytes 2024-09-13T09:42:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1576814592 bytes 2024-09-13T09:42:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1598822400 bytes 2024-09-13T09:42:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1657309184 bytes 2024-09-13T09:42:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1692843008 bytes 2024-09-13T09:42:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1709920256 bytes 2024-09-13T09:42:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1720872960 bytes 2024-09-13T09:42:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1746325504 bytes 2024-09-13T09:42:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1808697344 bytes 2024-09-13T09:42:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1862215680 bytes 2024-09-13T09:42:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1908786176 bytes 2024-09-13T09:42:14.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1939768320 bytes 2024-09-13T09:42:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2033116160 bytes 2024-09-13T09:42:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2060054528 bytes 2024-09-13T09:42:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2129291264 bytes 2024-09-13T09:42:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2144327680 bytes 2024-09-13T09:42:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2181049344 bytes 2024-09-13T09:42:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2235578368 bytes 2024-09-13T09:42:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2242960384 bytes 2024-09-13T09:42:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2308544512 bytes 2024-09-13T09:42:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2329530368 bytes 2024-09-13T09:42:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2383915008 bytes 2024-09-13T09:42:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2387053568 bytes 2024-09-13T09:42:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2418515968 bytes 2024-09-13T09:42:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2427975680 bytes 2024-09-13T09:42:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2432896000 bytes 2024-09-13T09:42:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2452006912 bytes 2024-09-13T09:42:16.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2487649280 bytes 2024-09-13T09:42:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2498116608 bytes 2024-09-13T09:42:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2533745664 bytes 2024-09-13T09:42:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2537835520 bytes 2024-09-13T09:42:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2583974912 bytes 2024-09-13T09:42:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2598239232 bytes 2024-09-13T09:42:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2619102208 bytes 2024-09-13T09:42:17.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2644353024 bytes 2024-09-13T09:42:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2723724288 bytes 2024-09-13T09:42:17.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2768591872 bytes 2024-09-13T09:42:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2776965120 bytes 2024-09-13T09:42:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:42:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:42:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:42:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:42:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:42:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_38 2024-09-13T09:42:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2024-09-13T09:42:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:42:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:42:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:42:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:42:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:42:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_38 128M 2024-09-13T09:42:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:42:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:42:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:42:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:42:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:42:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:42:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_38 2024-09-13T09:42:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:42:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2024-09-13T09:42:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:42:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:42:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:42:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:42:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2024-09-13T09:42:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:42:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2024-09-13T09:42:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_38': 2024-09-13T09:42:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:42:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:42:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:42:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 14054c8c9ad9 2024-09-13T09:42:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.14054c8c9ad9 2024-09-13T09:42:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:42:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:42:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:42:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:42:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:42:17 2024 2024-09-13T09:42:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:42:17 2024 2024-09-13T09:42:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:42:17 2024 2024-09-13T09:42:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:42:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:42:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 53855d34-b4b5-4803-8724-2964e4fd7b9d 2024-09-13T09:42:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:42:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:42:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_38 2024-09-13T09:42:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:42:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:42:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:42:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:42:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=8 2024-09-13T09:42:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:42:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:42:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 24403968 bytes 2024-09-13T09:42:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 36539392 bytes 2024-09-13T09:42:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 42169344 bytes 2024-09-13T09:42:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 65930240 bytes 2024-09-13T09:42:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 66545664 bytes 2024-09-13T09:42:19.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 133018624 bytes 2024-09-13T09:42:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 188173312 bytes 2024-09-13T09:42:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 252934144 bytes 2024-09-13T09:42:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 279187456 bytes 2024-09-13T09:42:19.827 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2024-09-13T09:42:19.825+0000 7f00cac16640 -1 reset not still connected to 0x55a18b99c410 2024-09-13T09:42:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 312412160 bytes 2024-09-13T09:42:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 361121792 bytes 2024-09-13T09:42:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 382103552 bytes 2024-09-13T09:42:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 449681408 bytes 2024-09-13T09:42:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 461598720 bytes 2024-09-13T09:42:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 506806272 bytes 2024-09-13T09:42:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 530706432 bytes 2024-09-13T09:42:20.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 576212992 bytes 2024-09-13T09:42:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 625790976 bytes 2024-09-13T09:42:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 693784576 bytes 2024-09-13T09:42:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 735179776 bytes 2024-09-13T09:42:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 837834752 bytes 2024-09-13T09:42:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 860183552 bytes 2024-09-13T09:42:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 887915520 bytes 2024-09-13T09:42:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 926765056 bytes 2024-09-13T09:42:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 984471552 bytes 2024-09-13T09:42:22.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 992712704 bytes 2024-09-13T09:42:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1024512000 bytes 2024-09-13T09:42:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1044755456 bytes 2024-09-13T09:42:22.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1088884736 bytes 2024-09-13T09:42:22.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1113018368 bytes 2024-09-13T09:42:22.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1113609216 bytes 2024-09-13T09:42:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1163984896 bytes 2024-09-13T09:42:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1197252608 bytes 2024-09-13T09:42:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1232223232 bytes 2024-09-13T09:42:23.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1232843776 bytes 2024-09-13T09:42:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1260151808 bytes 2024-09-13T09:42:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1278830592 bytes 2024-09-13T09:42:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1365071872 bytes 2024-09-13T09:42:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1395382272 bytes 2024-09-13T09:42:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1487019008 bytes 2024-09-13T09:42:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1510497280 bytes 2024-09-13T09:42:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1516608512 bytes 2024-09-13T09:42:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1543684096 bytes 2024-09-13T09:42:24.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1551461376 bytes 2024-09-13T09:42:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1578795008 bytes 2024-09-13T09:42:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1675171840 bytes 2024-09-13T09:42:24.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1762697216 bytes 2024-09-13T09:42:24.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1818717184 bytes 2024-09-13T09:42:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1819345920 bytes 2024-09-13T09:42:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1845476352 bytes 2024-09-13T09:42:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 1908214784 bytes 2024-09-13T09:42:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 1940603904 bytes 2024-09-13T09:42:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 1960506368 bytes 2024-09-13T09:42:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2000869376 bytes 2024-09-13T09:42:25.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2054634496 bytes 2024-09-13T09:42:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2086012928 bytes 2024-09-13T09:42:25.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2125208576 bytes 2024-09-13T09:42:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2162304000 bytes 2024-09-13T09:42:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2248987648 bytes 2024-09-13T09:42:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2304796672 bytes 2024-09-13T09:42:26.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2336114688 bytes 2024-09-13T09:42:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2399127552 bytes 2024-09-13T09:42:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2444846080 bytes 2024-09-13T09:42:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:42:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:42:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:42:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:42:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:42:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_39 2024-09-13T09:42:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2024-09-13T09:42:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:42:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:42:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:42:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:42:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:42:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_39 128M 2024-09-13T09:42:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:42:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:42:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:42:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:42:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:42:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:42:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_39 2024-09-13T09:42:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:42:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2024-09-13T09:42:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:42:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:42:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:42:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:42:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2024-09-13T09:42:27.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:42:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2024-09-13T09:42:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_39': 2024-09-13T09:42:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:42:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:42:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:42:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1411227c6ca 2024-09-13T09:42:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1411227c6ca 2024-09-13T09:42:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:42:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:42:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:42:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:42:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:42:26 2024 2024-09-13T09:42:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:42:26 2024 2024-09-13T09:42:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:42:26 2024 2024-09-13T09:42:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:42:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:42:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: f2864223-3140-4220-a2fa-798f1e72b93d 2024-09-13T09:42:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:42:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:42:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_39 2024-09-13T09:42:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:42:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:42:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:42:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:42:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=10 2024-09-13T09:42:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:42:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:42:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 28675072 bytes 2024-09-13T09:42:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 67798016 bytes 2024-09-13T09:42:28.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 68407296 bytes 2024-09-13T09:42:28.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 124736512 bytes 2024-09-13T09:42:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 175770624 bytes 2024-09-13T09:42:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 211353600 bytes 2024-09-13T09:42:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 231432192 bytes 2024-09-13T09:42:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 298404864 bytes 2024-09-13T09:42:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 372793344 bytes 2024-09-13T09:42:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 459948032 bytes 2024-09-13T09:42:29.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 488269824 bytes 2024-09-13T09:42:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 561290240 bytes 2024-09-13T09:42:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 637324288 bytes 2024-09-13T09:42:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 637919232 bytes 2024-09-13T09:42:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 697126912 bytes 2024-09-13T09:42:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 762697728 bytes 2024-09-13T09:42:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 799046656 bytes 2024-09-13T09:42:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 818836480 bytes 2024-09-13T09:42:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 863159296 bytes 2024-09-13T09:42:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 884196352 bytes 2024-09-13T09:42:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 908495872 bytes 2024-09-13T09:42:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 920267776 bytes 2024-09-13T09:42:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 969284608 bytes 2024-09-13T09:42:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 989830144 bytes 2024-09-13T09:42:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1044135936 bytes 2024-09-13T09:42:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1120704512 bytes 2024-09-13T09:42:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1157623808 bytes 2024-09-13T09:42:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1196517376 bytes 2024-09-13T09:42:31.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1221856256 bytes 2024-09-13T09:42:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1264684032 bytes 2024-09-13T09:42:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1333529600 bytes 2024-09-13T09:42:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1357373440 bytes 2024-09-13T09:42:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1371763712 bytes 2024-09-13T09:42:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1459670016 bytes 2024-09-13T09:42:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1487576064 bytes 2024-09-13T09:42:32.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1505595392 bytes 2024-09-13T09:42:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1527521280 bytes 2024-09-13T09:42:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1617465344 bytes 2024-09-13T09:42:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1747321856 bytes 2024-09-13T09:42:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1776929792 bytes 2024-09-13T09:42:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1840348160 bytes 2024-09-13T09:42:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1879832576 bytes 2024-09-13T09:42:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1921348608 bytes 2024-09-13T09:42:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1965608960 bytes 2024-09-13T09:42:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2010999808 bytes 2024-09-13T09:42:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2094604288 bytes 2024-09-13T09:42:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2121899008 bytes 2024-09-13T09:42:34.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2209975296 bytes 2024-09-13T09:42:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2270629888 bytes 2024-09-13T09:42:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2334731264 bytes 2024-09-13T09:42:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2388954112 bytes 2024-09-13T09:42:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2420176896 bytes 2024-09-13T09:42:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2420799488 bytes 2024-09-13T09:42:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2459112448 bytes 2024-09-13T09:42:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2459691008 bytes 2024-09-13T09:42:35.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2521746432 bytes 2024-09-13T09:42:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2560786432 bytes 2024-09-13T09:42:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2637874176 bytes 2024-09-13T09:42:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2683043840 bytes 2024-09-13T09:42:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2752312320 bytes 2024-09-13T09:42:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2815703040 bytes 2024-09-13T09:42:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2896807936 bytes 2024-09-13T09:42:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2967601152 bytes 2024-09-13T09:42:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2996055040 bytes 2024-09-13T09:42:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 3100271616 bytes 2024-09-13T09:42:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 3161548800 bytes 2024-09-13T09:42:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 3191287808 bytes 2024-09-13T09:42:36.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 3204996096 bytes 2024-09-13T09:42:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 3224887296 bytes 2024-09-13T09:42:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 3299179520 bytes 2024-09-13T09:42:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 3373272064 bytes 2024-09-13T09:42:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 3453375488 bytes 2024-09-13T09:42:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 3476545536 bytes 2024-09-13T09:42:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3481977856 bytes 2024-09-13T09:42:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:42:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:42:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:42:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:42:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:42:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_40 2024-09-13T09:42:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2024-09-13T09:42:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:42:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:42:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:42:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:42:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:42:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_40 128M 2024-09-13T09:42:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:42:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:42:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:42:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:42:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:42:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:42:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_40 2024-09-13T09:42:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:42:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2024-09-13T09:42:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:42:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:42:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:42:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:42:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2024-09-13T09:42:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:42:38.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2024-09-13T09:42:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_40': 2024-09-13T09:42:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:42:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:42:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:42:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 141de582e43f 2024-09-13T09:42:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.141de582e43f 2024-09-13T09:42:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:42:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:42:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:42:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:42:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:42:37 2024 2024-09-13T09:42:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:42:37 2024 2024-09-13T09:42:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:42:37 2024 2024-09-13T09:42:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:42:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:42:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 80af752a-a561-44aa-a889-99b950f68127 2024-09-13T09:42:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:42:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:42:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_40 2024-09-13T09:42:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:42:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:42:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:42:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:42:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2024-09-13T09:42:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:42:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:42:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 80233472 bytes 2024-09-13T09:42:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 97509376 bytes 2024-09-13T09:42:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 149731328 bytes 2024-09-13T09:42:39.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 167360512 bytes 2024-09-13T09:42:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 233357312 bytes 2024-09-13T09:42:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 282194944 bytes 2024-09-13T09:42:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 307897344 bytes 2024-09-13T09:42:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 366084096 bytes 2024-09-13T09:42:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 366686208 bytes 2024-09-13T09:42:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 367296512 bytes 2024-09-13T09:42:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 444517376 bytes 2024-09-13T09:42:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 497786880 bytes 2024-09-13T09:42:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 553950208 bytes 2024-09-13T09:42:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 579297280 bytes 2024-09-13T09:42:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 623428608 bytes 2024-09-13T09:42:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 663926784 bytes 2024-09-13T09:42:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 728921088 bytes 2024-09-13T09:42:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 804310016 bytes 2024-09-13T09:42:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 823473152 bytes 2024-09-13T09:42:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 902133760 bytes 2024-09-13T09:42:41.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 965067776 bytes 2024-09-13T09:42:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 986715136 bytes 2024-09-13T09:42:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1018043392 bytes 2024-09-13T09:42:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1056939008 bytes 2024-09-13T09:42:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1083553792 bytes 2024-09-13T09:42:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1113859072 bytes 2024-09-13T09:42:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1194597376 bytes 2024-09-13T09:42:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1225324544 bytes 2024-09-13T09:42:42.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1237603328 bytes 2024-09-13T09:42:42.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1277516800 bytes 2024-09-13T09:42:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1278129152 bytes 2024-09-13T09:42:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1333814272 bytes 2024-09-13T09:42:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1347450880 bytes 2024-09-13T09:42:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1365527552 bytes 2024-09-13T09:42:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1382824960 bytes 2024-09-13T09:42:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1461103616 bytes 2024-09-13T09:42:43.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1564949504 bytes 2024-09-13T09:42:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1593616384 bytes 2024-09-13T09:42:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1642061824 bytes 2024-09-13T09:42:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1656210432 bytes 2024-09-13T09:42:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1714712576 bytes 2024-09-13T09:42:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1761537024 bytes 2024-09-13T09:42:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1830236160 bytes 2024-09-13T09:42:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1881010176 bytes 2024-09-13T09:42:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1905193984 bytes 2024-09-13T09:42:44.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1989651456 bytes 2024-09-13T09:42:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2051848192 bytes 2024-09-13T09:42:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2069688320 bytes 2024-09-13T09:42:45.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2080714752 bytes 2024-09-13T09:42:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2118508544 bytes 2024-09-13T09:42:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2123365376 bytes 2024-09-13T09:42:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2136894464 bytes 2024-09-13T09:42:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2225563648 bytes 2024-09-13T09:42:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2276678656 bytes 2024-09-13T09:42:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2304687104 bytes 2024-09-13T09:42:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2305299456 bytes 2024-09-13T09:42:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2347046912 bytes 2024-09-13T09:42:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2434271232 bytes 2024-09-13T09:42:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2477294592 bytes 2024-09-13T09:42:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2494363648 bytes 2024-09-13T09:42:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2517574656 bytes 2024-09-13T09:42:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2557855744 bytes 2024-09-13T09:42:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2602914816 bytes 2024-09-13T09:42:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2643849216 bytes 2024-09-13T09:42:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2709539840 bytes 2024-09-13T09:42:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2786234368 bytes 2024-09-13T09:42:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2858824704 bytes 2024-09-13T09:42:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2859462656 bytes 2024-09-13T09:42:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2892812288 bytes 2024-09-13T09:42:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2948770816 bytes 2024-09-13T09:42:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2984938496 bytes 2024-09-13T09:42:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 3015083008 bytes 2024-09-13T09:42:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 3045792768 bytes 2024-09-13T09:42:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3046414336 bytes 2024-09-13T09:42:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3155083264 bytes 2024-09-13T09:42:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3190729728 bytes 2024-09-13T09:42:48.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3214804992 bytes 2024-09-13T09:42:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3252244480 bytes 2024-09-13T09:42:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3292278784 bytes 2024-09-13T09:42:49.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3363986432 bytes 2024-09-13T09:42:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3457589248 bytes 2024-09-13T09:42:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3534288896 bytes 2024-09-13T09:42:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3567553536 bytes 2024-09-13T09:42:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3592929280 bytes 2024-09-13T09:42:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3630255104 bytes 2024-09-13T09:42:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3712083968 bytes 2024-09-13T09:42:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3776300032 bytes 2024-09-13T09:42:50.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3809167360 bytes 2024-09-13T09:42:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3830533120 bytes 2024-09-13T09:42:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 3877302272 bytes 2024-09-13T09:42:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:42:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:42:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:42:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:42:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:42:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_41 2024-09-13T09:42:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2024-09-13T09:42:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:42:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:42:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:42:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:42:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:42:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_41 128M 2024-09-13T09:42:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:42:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:42:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:42:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:42:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:42:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:42:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:42:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_41 2024-09-13T09:42:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:42:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2024-09-13T09:42:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:42:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:42:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:42:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:42:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2024-09-13T09:42:51.409 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-13T09:42:51.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-13T09:42:51.410 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-13T09:42:51.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:42:51.409+0000 7f4b882fa640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 47481) UID: 1000 2024-09-13T09:42:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:42:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2024-09-13T09:42:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_41': 2024-09-13T09:42:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:42:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:42:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:42:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 14295625635d 2024-09-13T09:42:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.14295625635d 2024-09-13T09:42:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:42:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:42:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:42:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:42:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:42:50 2024 2024-09-13T09:42:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:42:50 2024 2024-09-13T09:42:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:42:50 2024 2024-09-13T09:42:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:42:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:42:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 80c5a437-de46-4aee-a1dd-7b12f06d8126 2024-09-13T09:42:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:42:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:42:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_41 2024-09-13T09:42:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:42:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:42:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:42:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:42:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=9 2024-09-13T09:42:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:42:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:42:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 11631616 bytes 2024-09-13T09:42:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 37766144 bytes 2024-09-13T09:42:52.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 49271808 bytes 2024-09-13T09:42:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 68291584 bytes 2024-09-13T09:42:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 81072128 bytes 2024-09-13T09:42:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 132875264 bytes 2024-09-13T09:42:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 171698176 bytes 2024-09-13T09:42:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 209768448 bytes 2024-09-13T09:42:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 234605568 bytes 2024-09-13T09:42:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 241778688 bytes 2024-09-13T09:42:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 310065152 bytes 2024-09-13T09:42:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 335193088 bytes 2024-09-13T09:42:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 360330240 bytes 2024-09-13T09:42:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 374035456 bytes 2024-09-13T09:42:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 450859008 bytes 2024-09-13T09:42:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 481417216 bytes 2024-09-13T09:42:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 485675008 bytes 2024-09-13T09:42:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 488876032 bytes 2024-09-13T09:42:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 526456832 bytes 2024-09-13T09:42:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 537012224 bytes 2024-09-13T09:42:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 589430784 bytes 2024-09-13T09:42:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 637612032 bytes 2024-09-13T09:42:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 667091968 bytes 2024-09-13T09:42:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 684037120 bytes 2024-09-13T09:42:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 711189504 bytes 2024-09-13T09:42:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 729123840 bytes 2024-09-13T09:42:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 751981568 bytes 2024-09-13T09:42:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 774535168 bytes 2024-09-13T09:42:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 792832000 bytes 2024-09-13T09:42:55.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 809162752 bytes 2024-09-13T09:42:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 860250112 bytes 2024-09-13T09:42:55.411 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-13T09:42:55.411 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:42:55.411 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-13T09:42:55.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-13T09:42:55.412 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-13T09:42:55.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-13T09:42:55.415 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-09-13T09:42:55.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 869291008 bytes 2024-09-13T09:42:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 916648960 bytes 2024-09-13T09:42:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 962476032 bytes 2024-09-13T09:42:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1042832384 bytes 2024-09-13T09:42:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1121275904 bytes 2024-09-13T09:42:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1169903616 bytes 2024-09-13T09:42:56.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1216621568 bytes 2024-09-13T09:42:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1299175424 bytes 2024-09-13T09:42:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1334033408 bytes 2024-09-13T09:42:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1334659072 bytes 2024-09-13T09:42:56.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1343581184 bytes 2024-09-13T09:42:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1418593280 bytes 2024-09-13T09:42:57.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1452092416 bytes 2024-09-13T09:42:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1469316096 bytes 2024-09-13T09:42:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1510770688 bytes 2024-09-13T09:42:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1554701312 bytes 2024-09-13T09:42:57.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1566818304 bytes 2024-09-13T09:42:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1611265024 bytes 2024-09-13T09:42:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1691212800 bytes 2024-09-13T09:42:58.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 1740859392 bytes 2024-09-13T09:42:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 1773399040 bytes 2024-09-13T09:42:58.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 1809773568 bytes 2024-09-13T09:42:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 1850642432 bytes 2024-09-13T09:42:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 1860939776 bytes 2024-09-13T09:42:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 1888315392 bytes 2024-09-13T09:42:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 1929545728 bytes 2024-09-13T09:42:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 1983818752 bytes 2024-09-13T09:42:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2044767232 bytes 2024-09-13T09:42:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2052030464 bytes 2024-09-13T09:42:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2113546240 bytes 2024-09-13T09:42:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2190645248 bytes 2024-09-13T09:42:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2202982400 bytes 2024-09-13T09:42:59.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2214678528 bytes 2024-09-13T09:42:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2273399808 bytes 2024-09-13T09:43:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2379342848 bytes 2024-09-13T09:43:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2461234176 bytes 2024-09-13T09:43:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2485324800 bytes 2024-09-13T09:43:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2506854400 bytes 2024-09-13T09:43:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2542171136 bytes 2024-09-13T09:43:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2563298304 bytes 2024-09-13T09:43:00.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:43:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:43:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:43:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:43:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:43:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_42 2024-09-13T09:43:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2024-09-13T09:43:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:43:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:43:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:43:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_42 128M 2024-09-13T09:43:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:43:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:43:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:43:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:43:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_42 2024-09-13T09:43:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:43:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2024-09-13T09:43:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:43:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:43:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2024-09-13T09:43:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:43:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2024-09-13T09:43:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_42': 2024-09-13T09:43:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:43:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:43:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:43:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 143dd5a3e168 2024-09-13T09:43:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.143dd5a3e168 2024-09-13T09:43:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:43:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:43:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:43:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:43:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:43:00 2024 2024-09-13T09:43:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:43:00 2024 2024-09-13T09:43:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:43:00 2024 2024-09-13T09:43:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:43:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:43:01.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 66db570e-e124-4bcd-9c58-ccbd57311102 2024-09-13T09:43:01.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:43:01.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:43:01.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_42 2024-09-13T09:43:01.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:01.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:01.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:43:01.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:43:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2024-09-13T09:43:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:43:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:43:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 20314112 bytes 2024-09-13T09:43:02.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 93038592 bytes 2024-09-13T09:43:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 141919232 bytes 2024-09-13T09:43:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 161715200 bytes 2024-09-13T09:43:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 207744000 bytes 2024-09-13T09:43:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 238513152 bytes 2024-09-13T09:43:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 332680192 bytes 2024-09-13T09:43:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 355741696 bytes 2024-09-13T09:43:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 398251008 bytes 2024-09-13T09:43:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 446054400 bytes 2024-09-13T09:43:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 521120768 bytes 2024-09-13T09:43:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 595143680 bytes 2024-09-13T09:43:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 628258816 bytes 2024-09-13T09:43:03.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 642311168 bytes 2024-09-13T09:43:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 689594368 bytes 2024-09-13T09:43:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 768522240 bytes 2024-09-13T09:43:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 816766976 bytes 2024-09-13T09:43:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 883258368 bytes 2024-09-13T09:43:04.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 908201984 bytes 2024-09-13T09:43:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 976344064 bytes 2024-09-13T09:43:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1031770112 bytes 2024-09-13T09:43:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1104636928 bytes 2024-09-13T09:43:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1145698304 bytes 2024-09-13T09:43:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1158180864 bytes 2024-09-13T09:43:05.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1183821824 bytes 2024-09-13T09:43:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1254824960 bytes 2024-09-13T09:43:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1279384576 bytes 2024-09-13T09:43:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1357097984 bytes 2024-09-13T09:43:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1378215936 bytes 2024-09-13T09:43:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1387715584 bytes 2024-09-13T09:43:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1443355648 bytes 2024-09-13T09:43:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1446285312 bytes 2024-09-13T09:43:06.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1478668288 bytes 2024-09-13T09:43:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1490525184 bytes 2024-09-13T09:43:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1525903360 bytes 2024-09-13T09:43:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1531304960 bytes 2024-09-13T09:43:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1575733248 bytes 2024-09-13T09:43:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1618771968 bytes 2024-09-13T09:43:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1657136128 bytes 2024-09-13T09:43:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1693591552 bytes 2024-09-13T09:43:06.917 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2024-09-13T09:43:06.912+0000 7ff85b706640 -1 reset not still connected to 0x5568f2580270 2024-09-13T09:43:06.917 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2024-09-13T09:43:06.912+0000 7ff85b706640 -1 reset not still connected to 0x5568f3051ee0 2024-09-13T09:43:06.917 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2024-09-13T09:43:06.912+0000 7ff85b706640 -1 reset not still connected to 0x5568fa148ea0 2024-09-13T09:43:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1818936320 bytes 2024-09-13T09:43:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1840454656 bytes 2024-09-13T09:43:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1874990080 bytes 2024-09-13T09:43:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1901242368 bytes 2024-09-13T09:43:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1911668736 bytes 2024-09-13T09:43:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1942521856 bytes 2024-09-13T09:43:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1943134208 bytes 2024-09-13T09:43:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1997511680 bytes 2024-09-13T09:43:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2078789632 bytes 2024-09-13T09:43:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2120605696 bytes 2024-09-13T09:43:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2214549504 bytes 2024-09-13T09:43:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2258215936 bytes 2024-09-13T09:43:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2258814976 bytes 2024-09-13T09:43:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2281568256 bytes 2024-09-13T09:43:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2320144384 bytes 2024-09-13T09:43:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2407224320 bytes 2024-09-13T09:43:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2444816384 bytes 2024-09-13T09:43:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2462282752 bytes 2024-09-13T09:43:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2481476608 bytes 2024-09-13T09:43:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2500250624 bytes 2024-09-13T09:43:09.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2607924224 bytes 2024-09-13T09:43:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2696094720 bytes 2024-09-13T09:43:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2721651712 bytes 2024-09-13T09:43:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2733972480 bytes 2024-09-13T09:43:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2769794048 bytes 2024-09-13T09:43:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2858177536 bytes 2024-09-13T09:43:10.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2894655488 bytes 2024-09-13T09:43:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2961870848 bytes 2024-09-13T09:43:10.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 3000480768 bytes 2024-09-13T09:43:11.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 3090804736 bytes 2024-09-13T09:43:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 3127394304 bytes 2024-09-13T09:43:11.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 3129093120 bytes 2024-09-13T09:43:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 3164030976 bytes 2024-09-13T09:43:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3229219840 bytes 2024-09-13T09:43:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3337089024 bytes 2024-09-13T09:43:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3358726144 bytes 2024-09-13T09:43:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3366672384 bytes 2024-09-13T09:43:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3402930176 bytes 2024-09-13T09:43:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3449913344 bytes 2024-09-13T09:43:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3459207168 bytes 2024-09-13T09:43:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3504422912 bytes 2024-09-13T09:43:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3556875264 bytes 2024-09-13T09:43:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3612400640 bytes 2024-09-13T09:43:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3659873280 bytes 2024-09-13T09:43:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3681768448 bytes 2024-09-13T09:43:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3695985664 bytes 2024-09-13T09:43:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3727856640 bytes 2024-09-13T09:43:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3728454656 bytes 2024-09-13T09:43:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3741875200 bytes 2024-09-13T09:43:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 3765729280 bytes 2024-09-13T09:43:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 3839287296 bytes 2024-09-13T09:43:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 3876747264 bytes 2024-09-13T09:43:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 3917005824 bytes 2024-09-13T09:43:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:43:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:43:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:43:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:43:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:43:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_43 2024-09-13T09:43:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2024-09-13T09:43:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:43:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:43:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:43:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_43 128M 2024-09-13T09:43:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:43:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:43:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:43:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:43:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_43 2024-09-13T09:43:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:43:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2024-09-13T09:43:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:43:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:43:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2024-09-13T09:43:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:43:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2024-09-13T09:43:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_43': 2024-09-13T09:43:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:43:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:43:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:43:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1449f104cc01 2024-09-13T09:43:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1449f104cc01 2024-09-13T09:43:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:43:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:43:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:43:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:43:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:43:13 2024 2024-09-13T09:43:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:43:13 2024 2024-09-13T09:43:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:43:13 2024 2024-09-13T09:43:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:43:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:43:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 69d778b5-940c-4aa1-8021-ea2d132d002e 2024-09-13T09:43:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:43:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:43:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_43 2024-09-13T09:43:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:43:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:43:14.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=9 2024-09-13T09:43:14.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:43:14.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:43:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 45522944 bytes 2024-09-13T09:43:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 104344576 bytes 2024-09-13T09:43:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 148629504 bytes 2024-09-13T09:43:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 212345856 bytes 2024-09-13T09:43:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 244605952 bytes 2024-09-13T09:43:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 284160000 bytes 2024-09-13T09:43:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 340094976 bytes 2024-09-13T09:43:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 363014144 bytes 2024-09-13T09:43:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 363618304 bytes 2024-09-13T09:43:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 375787520 bytes 2024-09-13T09:43:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 420463616 bytes 2024-09-13T09:43:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 435238912 bytes 2024-09-13T09:43:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 468659200 bytes 2024-09-13T09:43:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 554803200 bytes 2024-09-13T09:43:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 630732800 bytes 2024-09-13T09:43:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 677932032 bytes 2024-09-13T09:43:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 722704384 bytes 2024-09-13T09:43:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 754848768 bytes 2024-09-13T09:43:17.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 800563200 bytes 2024-09-13T09:43:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 902596608 bytes 2024-09-13T09:43:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 992317440 bytes 2024-09-13T09:43:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1069081600 bytes 2024-09-13T09:43:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1113495552 bytes 2024-09-13T09:43:18.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1155348480 bytes 2024-09-13T09:43:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1245175808 bytes 2024-09-13T09:43:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1294154752 bytes 2024-09-13T09:43:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1316832256 bytes 2024-09-13T09:43:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1332475904 bytes 2024-09-13T09:43:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1333063680 bytes 2024-09-13T09:43:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1398385664 bytes 2024-09-13T09:43:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1478016000 bytes 2024-09-13T09:43:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1500782592 bytes 2024-09-13T09:43:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1572897792 bytes 2024-09-13T09:43:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1634836480 bytes 2024-09-13T09:43:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1643421696 bytes 2024-09-13T09:43:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1704936448 bytes 2024-09-13T09:43:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1780892672 bytes 2024-09-13T09:43:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1837891584 bytes 2024-09-13T09:43:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1957609472 bytes 2024-09-13T09:43:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 2028359680 bytes 2024-09-13T09:43:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 2033239040 bytes 2024-09-13T09:43:20.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 2073626624 bytes 2024-09-13T09:43:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 2109246464 bytes 2024-09-13T09:43:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2157467648 bytes 2024-09-13T09:43:21.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2342551552 bytes 2024-09-13T09:43:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2418874368 bytes 2024-09-13T09:43:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2513978368 bytes 2024-09-13T09:43:21.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2559821824 bytes 2024-09-13T09:43:21.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2613329920 bytes 2024-09-13T09:43:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2658788352 bytes 2024-09-13T09:43:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2699692032 bytes 2024-09-13T09:43:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2739243008 bytes 2024-09-13T09:43:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2770414592 bytes 2024-09-13T09:43:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2805876736 bytes 2024-09-13T09:43:22.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2881934336 bytes 2024-09-13T09:43:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2904856576 bytes 2024-09-13T09:43:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2935198720 bytes 2024-09-13T09:43:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2977638400 bytes 2024-09-13T09:43:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 3016413184 bytes 2024-09-13T09:43:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 3079607296 bytes 2024-09-13T09:43:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 3108182016 bytes 2024-09-13T09:43:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 3108808704 bytes 2024-09-13T09:43:23.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 3138444288 bytes 2024-09-13T09:43:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:43:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:43:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:43:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:43:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:43:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_44 2024-09-13T09:43:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2024-09-13T09:43:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:43:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:43:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:43:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_44 128M 2024-09-13T09:43:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:43:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:43:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:43:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:43:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_44 2024-09-13T09:43:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:43:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2024-09-13T09:43:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:43:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:43:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2024-09-13T09:43:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:43:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2024-09-13T09:43:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_44': 2024-09-13T09:43:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:43:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:43:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:43:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1455849aa79e 2024-09-13T09:43:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1455849aa79e 2024-09-13T09:43:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:43:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:43:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:43:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:43:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:43:23 2024 2024-09-13T09:43:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:43:23 2024 2024-09-13T09:43:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:43:23 2024 2024-09-13T09:43:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:43:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:43:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 90e5e796-ce61-474c-b054-7f1a077502a8 2024-09-13T09:43:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:43:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:43:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_44 2024-09-13T09:43:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:43:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:43:24.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=8 2024-09-13T09:43:24.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:43:24.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:43:25.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 107916288 bytes 2024-09-13T09:43:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 204863488 bytes 2024-09-13T09:43:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 250312704 bytes 2024-09-13T09:43:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 290605056 bytes 2024-09-13T09:43:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 321401856 bytes 2024-09-13T09:43:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 341523456 bytes 2024-09-13T09:43:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 414456832 bytes 2024-09-13T09:43:26.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 449239040 bytes 2024-09-13T09:43:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 475186176 bytes 2024-09-13T09:43:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 518169600 bytes 2024-09-13T09:43:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 562276352 bytes 2024-09-13T09:43:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 587890688 bytes 2024-09-13T09:43:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 629230592 bytes 2024-09-13T09:43:26.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 648259584 bytes 2024-09-13T09:43:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 729828352 bytes 2024-09-13T09:43:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 748696576 bytes 2024-09-13T09:43:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 758775808 bytes 2024-09-13T09:43:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 792403968 bytes 2024-09-13T09:43:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 814326784 bytes 2024-09-13T09:43:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 834498560 bytes 2024-09-13T09:43:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 835113984 bytes 2024-09-13T09:43:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 835718144 bytes 2024-09-13T09:43:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 914033664 bytes 2024-09-13T09:43:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 964025344 bytes 2024-09-13T09:43:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1014264832 bytes 2024-09-13T09:43:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1031498752 bytes 2024-09-13T09:43:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1047444480 bytes 2024-09-13T09:43:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1109801984 bytes 2024-09-13T09:43:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1110437888 bytes 2024-09-13T09:43:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1126853632 bytes 2024-09-13T09:43:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1168489472 bytes 2024-09-13T09:43:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1173105664 bytes 2024-09-13T09:43:28.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1222171648 bytes 2024-09-13T09:43:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1266611200 bytes 2024-09-13T09:43:29.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1345890304 bytes 2024-09-13T09:43:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1430004736 bytes 2024-09-13T09:43:29.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1445331968 bytes 2024-09-13T09:43:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1491572736 bytes 2024-09-13T09:43:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1550630912 bytes 2024-09-13T09:43:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1567771648 bytes 2024-09-13T09:43:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1587730432 bytes 2024-09-13T09:43:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1661714432 bytes 2024-09-13T09:43:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1680316416 bytes 2024-09-13T09:43:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1739513856 bytes 2024-09-13T09:43:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1838824448 bytes 2024-09-13T09:43:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1891216384 bytes 2024-09-13T09:43:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1933862912 bytes 2024-09-13T09:43:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1934483456 bytes 2024-09-13T09:43:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2018061312 bytes 2024-09-13T09:43:31.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2037614592 bytes 2024-09-13T09:43:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2059362304 bytes 2024-09-13T09:43:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2102653952 bytes 2024-09-13T09:43:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2130483200 bytes 2024-09-13T09:43:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2151113728 bytes 2024-09-13T09:43:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2222592000 bytes 2024-09-13T09:43:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2236575744 bytes 2024-09-13T09:43:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2315559936 bytes 2024-09-13T09:43:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2390315008 bytes 2024-09-13T09:43:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2429004800 bytes 2024-09-13T09:43:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2500434944 bytes 2024-09-13T09:43:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2598542336 bytes 2024-09-13T09:43:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:43:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:43:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:43:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:43:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:43:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_45 2024-09-13T09:43:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2024-09-13T09:43:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:43:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:43:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:43:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_45 128M 2024-09-13T09:43:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:43:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:43:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:43:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:43:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_45 2024-09-13T09:43:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:43:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2024-09-13T09:43:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:43:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:43:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2024-09-13T09:43:33.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:43:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2024-09-13T09:43:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_45': 2024-09-13T09:43:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:43:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:43:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:43:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 14619f4c40bc 2024-09-13T09:43:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.14619f4c40bc 2024-09-13T09:43:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:43:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:43:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:43:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:43:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:43:32 2024 2024-09-13T09:43:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:43:32 2024 2024-09-13T09:43:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:43:32 2024 2024-09-13T09:43:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:43:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:43:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 24eb5df8-5de6-4058-952e-856c1cb902e7 2024-09-13T09:43:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:43:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:43:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_45 2024-09-13T09:43:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:43:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:43:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=10 2024-09-13T09:43:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:43:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:43:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 24085504 bytes 2024-09-13T09:43:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 77840384 bytes 2024-09-13T09:43:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 109273088 bytes 2024-09-13T09:43:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 158562304 bytes 2024-09-13T09:43:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 196570112 bytes 2024-09-13T09:43:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 236217344 bytes 2024-09-13T09:43:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 246156288 bytes 2024-09-13T09:43:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 297872384 bytes 2024-09-13T09:43:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 300989440 bytes 2024-09-13T09:43:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 331039744 bytes 2024-09-13T09:43:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 374463488 bytes 2024-09-13T09:43:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 409167872 bytes 2024-09-13T09:43:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 417680384 bytes 2024-09-13T09:43:35.519 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7246487547107812) > weight (0.25) 2024-09-13T09:43:35.519 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-13T09:43:35.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-13T09:43:35.520 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-13T09:43:35.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T09:43:35.519+0000 7f4d61bab640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 46264) UID: 1000 2024-09-13T09:43:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 445229056 bytes 2024-09-13T09:43:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 496400384 bytes 2024-09-13T09:43:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 522141696 bytes 2024-09-13T09:43:35.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 540242944 bytes 2024-09-13T09:43:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 610827264 bytes 2024-09-13T09:43:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 693976064 bytes 2024-09-13T09:43:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 721053696 bytes 2024-09-13T09:43:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 733216768 bytes 2024-09-13T09:43:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 768642048 bytes 2024-09-13T09:43:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 769263616 bytes 2024-09-13T09:43:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 772147200 bytes 2024-09-13T09:43:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 772760576 bytes 2024-09-13T09:43:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 789646336 bytes 2024-09-13T09:43:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 828275712 bytes 2024-09-13T09:43:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 934043648 bytes 2024-09-13T09:43:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 979494912 bytes 2024-09-13T09:43:37.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1000908800 bytes 2024-09-13T09:43:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1008745472 bytes 2024-09-13T09:43:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1034471424 bytes 2024-09-13T09:43:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1058059264 bytes 2024-09-13T09:43:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1164487680 bytes 2024-09-13T09:43:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1191746560 bytes 2024-09-13T09:43:38.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1269127168 bytes 2024-09-13T09:43:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1278086144 bytes 2024-09-13T09:43:38.521 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-13T09:43:38.521 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:43:38.521 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-13T09:43:38.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-13T09:43:38.522 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-13T09:43:38.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-13T09:43:38.525 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2024-09-13T09:43:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1324701696 bytes 2024-09-13T09:43:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1393007616 bytes 2024-09-13T09:43:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1459304448 bytes 2024-09-13T09:43:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1459921920 bytes 2024-09-13T09:43:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1503652864 bytes 2024-09-13T09:43:39.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1552496640 bytes 2024-09-13T09:43:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1615537152 bytes 2024-09-13T09:43:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1642435584 bytes 2024-09-13T09:43:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1662999552 bytes 2024-09-13T09:43:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1723233280 bytes 2024-09-13T09:43:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1783635968 bytes 2024-09-13T09:43:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1800924160 bytes 2024-09-13T09:43:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1825245184 bytes 2024-09-13T09:43:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 1855787008 bytes 2024-09-13T09:43:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 1867469824 bytes 2024-09-13T09:43:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 1941505024 bytes 2024-09-13T09:43:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2019697664 bytes 2024-09-13T09:43:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2044724224 bytes 2024-09-13T09:43:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2071117824 bytes 2024-09-13T09:43:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2116611072 bytes 2024-09-13T09:43:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2156448768 bytes 2024-09-13T09:43:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2166233088 bytes 2024-09-13T09:43:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2229441536 bytes 2024-09-13T09:43:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2284081152 bytes 2024-09-13T09:43:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2312377344 bytes 2024-09-13T09:43:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2439157760 bytes 2024-09-13T09:43:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2468762624 bytes 2024-09-13T09:43:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2504197120 bytes 2024-09-13T09:43:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2543303680 bytes 2024-09-13T09:43:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2582504448 bytes 2024-09-13T09:43:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2589716480 bytes 2024-09-13T09:43:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2685826048 bytes 2024-09-13T09:43:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2716726272 bytes 2024-09-13T09:43:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2778741760 bytes 2024-09-13T09:43:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2851193856 bytes 2024-09-13T09:43:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2931861504 bytes 2024-09-13T09:43:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 2956192768 bytes 2024-09-13T09:43:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3001860096 bytes 2024-09-13T09:43:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3049968640 bytes 2024-09-13T09:43:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:43:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:43:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:43:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:43:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:43:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_46 2024-09-13T09:43:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2024-09-13T09:43:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:43:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:43:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:43:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_46 128M 2024-09-13T09:43:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:43:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:43:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:43:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:43:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_46 2024-09-13T09:43:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:43:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2024-09-13T09:43:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:43:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:43:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2024-09-13T09:43:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:43:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2024-09-13T09:43:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_46': 2024-09-13T09:43:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:43:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:43:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:43:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1473e9a9f4a7 2024-09-13T09:43:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1473e9a9f4a7 2024-09-13T09:43:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:43:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:43:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:43:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:43:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:43:44 2024 2024-09-13T09:43:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:43:44 2024 2024-09-13T09:43:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:43:44 2024 2024-09-13T09:43:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:43:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:43:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: b2303ec4-1348-4438-91f9-318b0bbb03c1 2024-09-13T09:43:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:43:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:43:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_46 2024-09-13T09:43:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:43:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:43:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=7 2024-09-13T09:43:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:43:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:43:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 33238016 bytes 2024-09-13T09:43:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 66344960 bytes 2024-09-13T09:43:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 154237952 bytes 2024-09-13T09:43:45.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 247283712 bytes 2024-09-13T09:43:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 252585984 bytes 2024-09-13T09:43:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 315395072 bytes 2024-09-13T09:43:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 348428288 bytes 2024-09-13T09:43:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 382216192 bytes 2024-09-13T09:43:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 514211840 bytes 2024-09-13T09:43:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 545421312 bytes 2024-09-13T09:43:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 546029568 bytes 2024-09-13T09:43:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 580513792 bytes 2024-09-13T09:43:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 614987776 bytes 2024-09-13T09:43:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 655948800 bytes 2024-09-13T09:43:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 715748352 bytes 2024-09-13T09:43:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 759927808 bytes 2024-09-13T09:43:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 776481792 bytes 2024-09-13T09:43:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 807045120 bytes 2024-09-13T09:43:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 862475264 bytes 2024-09-13T09:43:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 910715904 bytes 2024-09-13T09:43:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 954549248 bytes 2024-09-13T09:43:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1021414400 bytes 2024-09-13T09:43:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1071311872 bytes 2024-09-13T09:43:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1111056384 bytes 2024-09-13T09:43:48.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1136314368 bytes 2024-09-13T09:43:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1139357696 bytes 2024-09-13T09:43:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1147482112 bytes 2024-09-13T09:43:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1193219072 bytes 2024-09-13T09:43:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1273925632 bytes 2024-09-13T09:43:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1327411200 bytes 2024-09-13T09:43:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1378044928 bytes 2024-09-13T09:43:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1380317184 bytes 2024-09-13T09:43:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1437339648 bytes 2024-09-13T09:43:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1501464576 bytes 2024-09-13T09:43:49.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1530386432 bytes 2024-09-13T09:43:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1553195008 bytes 2024-09-13T09:43:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1586665472 bytes 2024-09-13T09:43:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1612977152 bytes 2024-09-13T09:43:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1642011648 bytes 2024-09-13T09:43:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1661673472 bytes 2024-09-13T09:43:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1683333120 bytes 2024-09-13T09:43:50.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1738199040 bytes 2024-09-13T09:43:50.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1773520896 bytes 2024-09-13T09:43:50.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1808724992 bytes 2024-09-13T09:43:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1820174336 bytes 2024-09-13T09:43:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1831558144 bytes 2024-09-13T09:43:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1848252416 bytes 2024-09-13T09:43:50.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1865164800 bytes 2024-09-13T09:43:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1878389760 bytes 2024-09-13T09:43:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1973763072 bytes 2024-09-13T09:43:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2027247616 bytes 2024-09-13T09:43:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2038391808 bytes 2024-09-13T09:43:51.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2063746048 bytes 2024-09-13T09:43:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2158663680 bytes 2024-09-13T09:43:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2239289344 bytes 2024-09-13T09:43:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:43:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:43:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:43:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:43:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:43:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_47 2024-09-13T09:43:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2024-09-13T09:43:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:43:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:43:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:43:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_47 128M 2024-09-13T09:43:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:43:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:43:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:43:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:43:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_47 2024-09-13T09:43:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:43:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2024-09-13T09:43:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:43:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:43:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2024-09-13T09:43:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:43:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2024-09-13T09:43:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_47': 2024-09-13T09:43:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:43:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:43:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:43:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 147ff3a24e07 2024-09-13T09:43:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.147ff3a24e07 2024-09-13T09:43:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:43:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:43:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:43:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:43:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:43:51 2024 2024-09-13T09:43:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:43:51 2024 2024-09-13T09:43:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:43:51 2024 2024-09-13T09:43:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:43:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:43:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 89dad582-4466-4ae2-9cac-bfea345b1268 2024-09-13T09:43:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:43:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:43:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_47 2024-09-13T09:43:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:43:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:43:52.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=6 2024-09-13T09:43:52.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:43:52.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:43:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 19932160 bytes 2024-09-13T09:43:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 120742912 bytes 2024-09-13T09:43:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 194927616 bytes 2024-09-13T09:43:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 220726272 bytes 2024-09-13T09:43:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 273170432 bytes 2024-09-13T09:43:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 319295488 bytes 2024-09-13T09:43:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 351101952 bytes 2024-09-13T09:43:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 414004224 bytes 2024-09-13T09:43:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 460506112 bytes 2024-09-13T09:43:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 519215104 bytes 2024-09-13T09:43:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 552374272 bytes 2024-09-13T09:43:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 576283648 bytes 2024-09-13T09:43:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 608328704 bytes 2024-09-13T09:43:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 608924672 bytes 2024-09-13T09:43:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 639142912 bytes 2024-09-13T09:43:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 639723520 bytes 2024-09-13T09:43:55.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 684169216 bytes 2024-09-13T09:43:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 742219776 bytes 2024-09-13T09:43:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 764238848 bytes 2024-09-13T09:43:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 858576896 bytes 2024-09-13T09:43:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 880395264 bytes 2024-09-13T09:43:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 889819136 bytes 2024-09-13T09:43:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 948623360 bytes 2024-09-13T09:43:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 970636288 bytes 2024-09-13T09:43:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1008954368 bytes 2024-09-13T09:43:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1074048000 bytes 2024-09-13T09:43:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1118613504 bytes 2024-09-13T09:43:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1152455680 bytes 2024-09-13T09:43:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1185481728 bytes 2024-09-13T09:43:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1217380352 bytes 2024-09-13T09:43:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1273003008 bytes 2024-09-13T09:43:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1303142400 bytes 2024-09-13T09:43:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1367027712 bytes 2024-09-13T09:43:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1376195584 bytes 2024-09-13T09:43:57.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1397440512 bytes 2024-09-13T09:43:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1434246144 bytes 2024-09-13T09:43:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1465611264 bytes 2024-09-13T09:43:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1525793792 bytes 2024-09-13T09:43:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1583093760 bytes 2024-09-13T09:43:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1634796544 bytes 2024-09-13T09:43:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1688134656 bytes 2024-09-13T09:43:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1746405376 bytes 2024-09-13T09:43:58.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1765106688 bytes 2024-09-13T09:43:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1798178816 bytes 2024-09-13T09:43:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1829264384 bytes 2024-09-13T09:43:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:43:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:43:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:43:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:43:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:43:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_48 2024-09-13T09:43:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2024-09-13T09:43:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:43:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:43:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:43:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_48 128M 2024-09-13T09:43:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:43:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:43:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:43:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:43:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:43:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_48 2024-09-13T09:43:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:43:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2024-09-13T09:43:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:43:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:43:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2024-09-13T09:43:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:43:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2024-09-13T09:43:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_48': 2024-09-13T09:43:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:43:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:43:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:43:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 148b84c5a05d 2024-09-13T09:43:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.148b84c5a05d 2024-09-13T09:43:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:43:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:43:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:43:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:43:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:43:58 2024 2024-09-13T09:43:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:43:58 2024 2024-09-13T09:43:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:43:58 2024 2024-09-13T09:43:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:43:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:43:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 186c00be-d8ec-4acc-8296-30f83c1b7ec8 2024-09-13T09:43:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:43:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:43:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_48 2024-09-13T09:43:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:43:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:43:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:43:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:43:59.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=9 2024-09-13T09:43:59.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:43:59.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:44:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 38233088 bytes 2024-09-13T09:44:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 60328960 bytes 2024-09-13T09:44:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 102899712 bytes 2024-09-13T09:44:00.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 145821696 bytes 2024-09-13T09:44:00.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 146891776 bytes 2024-09-13T09:44:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 233061376 bytes 2024-09-13T09:44:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 294284288 bytes 2024-09-13T09:44:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 378787840 bytes 2024-09-13T09:44:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 450801664 bytes 2024-09-13T09:44:01.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 490881024 bytes 2024-09-13T09:44:01.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 560694272 bytes 2024-09-13T09:44:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 624969728 bytes 2024-09-13T09:44:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 642667520 bytes 2024-09-13T09:44:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 696126464 bytes 2024-09-13T09:44:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 740810752 bytes 2024-09-13T09:44:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 743346176 bytes 2024-09-13T09:44:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 791211008 bytes 2024-09-13T09:44:02.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 843181056 bytes 2024-09-13T09:44:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 899813376 bytes 2024-09-13T09:44:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 980860928 bytes 2024-09-13T09:44:02.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 1026974720 bytes 2024-09-13T09:44:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1069136896 bytes 2024-09-13T09:44:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1082004480 bytes 2024-09-13T09:44:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1102391296 bytes 2024-09-13T09:44:03.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1114601472 bytes 2024-09-13T09:44:03.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1121382400 bytes 2024-09-13T09:44:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1189386240 bytes 2024-09-13T09:44:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1200437248 bytes 2024-09-13T09:44:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1206200320 bytes 2024-09-13T09:44:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1235726336 bytes 2024-09-13T09:44:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1274133504 bytes 2024-09-13T09:44:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1303200768 bytes 2024-09-13T09:44:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1364253696 bytes 2024-09-13T09:44:04.416 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.41896206378460377) > weight (0.25) 2024-09-13T09:44:04.416 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-13T09:44:04.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-13T09:44:04.417 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-13T09:44:04.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:44:04.416+0000 7fa197990640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 40154) UID: 1000 2024-09-13T09:44:04.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1370412032 bytes 2024-09-13T09:44:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1459591168 bytes 2024-09-13T09:44:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1460218880 bytes 2024-09-13T09:44:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1492846592 bytes 2024-09-13T09:44:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1549279232 bytes 2024-09-13T09:44:05.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1582445568 bytes 2024-09-13T09:44:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1588408320 bytes 2024-09-13T09:44:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1672650752 bytes 2024-09-13T09:44:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1699539968 bytes 2024-09-13T09:44:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1728711680 bytes 2024-09-13T09:44:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1755369472 bytes 2024-09-13T09:44:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1836292096 bytes 2024-09-13T09:44:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1879198720 bytes 2024-09-13T09:44:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1950529536 bytes 2024-09-13T09:44:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1977614336 bytes 2024-09-13T09:44:06.418 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-13T09:44:06.419 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:44:06.419 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-13T09:44:06.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-13T09:44:06.419 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-13T09:44:06.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-13T09:44:06.422 INFO:tasks.rbd_mirror_thrash:waiting for 112 secs before thrashing 2024-09-13T09:44:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1997550592 bytes 2024-09-13T09:44:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2069196800 bytes 2024-09-13T09:44:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2155655168 bytes 2024-09-13T09:44:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2178437120 bytes 2024-09-13T09:44:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2203844608 bytes 2024-09-13T09:44:07.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2204486656 bytes 2024-09-13T09:44:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2250830848 bytes 2024-09-13T09:44:07.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2251423744 bytes 2024-09-13T09:44:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2336073728 bytes 2024-09-13T09:44:07.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2428887040 bytes 2024-09-13T09:44:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2470346752 bytes 2024-09-13T09:44:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2535374848 bytes 2024-09-13T09:44:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2609768448 bytes 2024-09-13T09:44:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2665915392 bytes 2024-09-13T09:44:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2685735936 bytes 2024-09-13T09:44:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2803659776 bytes 2024-09-13T09:44:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2816090112 bytes 2024-09-13T09:44:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2841190400 bytes 2024-09-13T09:44:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:44:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:44:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:44:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:44:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:44:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_49 2024-09-13T09:44:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2024-09-13T09:44:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:44:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:44:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:44:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:44:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:44:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:44:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:44:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:44:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:44:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_49 128M 2024-09-13T09:44:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:44:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:44:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:44:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:44:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:44:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:44:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:44:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:44:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:44:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:44:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_49 2024-09-13T09:44:08.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:44:08.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2024-09-13T09:44:08.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:44:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:44:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:44:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:44:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2024-09-13T09:44:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:44:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2024-09-13T09:44:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_49': 2024-09-13T09:44:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:44:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:44:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:44:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 14a058b098fc 2024-09-13T09:44:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.14a058b098fc 2024-09-13T09:44:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:44:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:44:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:44:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:44:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:44:08 2024 2024-09-13T09:44:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:44:08 2024 2024-09-13T09:44:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:44:08 2024 2024-09-13T09:44:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:44:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:44:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: a0524d8d-2e9d-4921-a7cd-cfc9cc8a519c 2024-09-13T09:44:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:44:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:44:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_49 2024-09-13T09:44:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:44:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:44:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:44:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:44:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=7 2024-09-13T09:44:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:44:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:44:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 30896128 bytes 2024-09-13T09:44:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 92740608 bytes 2024-09-13T09:44:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 171034624 bytes 2024-09-13T09:44:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 186854400 bytes 2024-09-13T09:44:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 225410048 bytes 2024-09-13T09:44:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 235054080 bytes 2024-09-13T09:44:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 298934272 bytes 2024-09-13T09:44:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 316265472 bytes 2024-09-13T09:44:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 319851520 bytes 2024-09-13T09:44:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 338358272 bytes 2024-09-13T09:44:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 338970624 bytes 2024-09-13T09:44:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 376154112 bytes 2024-09-13T09:44:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 411830272 bytes 2024-09-13T09:44:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 449143808 bytes 2024-09-13T09:44:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 449775616 bytes 2024-09-13T09:44:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 499230720 bytes 2024-09-13T09:44:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 567665664 bytes 2024-09-13T09:44:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 633969664 bytes 2024-09-13T09:44:12.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 650375168 bytes 2024-09-13T09:44:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 651005952 bytes 2024-09-13T09:44:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 745623552 bytes 2024-09-13T09:44:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 785676288 bytes 2024-09-13T09:44:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 786278400 bytes 2024-09-13T09:44:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 831300608 bytes 2024-09-13T09:44:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 857980928 bytes 2024-09-13T09:44:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 963288064 bytes 2024-09-13T09:44:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1009629184 bytes 2024-09-13T09:44:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1046290432 bytes 2024-09-13T09:44:13.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1056743424 bytes 2024-09-13T09:44:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1133358080 bytes 2024-09-13T09:44:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1146336256 bytes 2024-09-13T09:44:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1146929152 bytes 2024-09-13T09:44:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1165119488 bytes 2024-09-13T09:44:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1212296192 bytes 2024-09-13T09:44:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1230971904 bytes 2024-09-13T09:44:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1281379328 bytes 2024-09-13T09:44:14.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1321914368 bytes 2024-09-13T09:44:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1371637760 bytes 2024-09-13T09:44:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1465490432 bytes 2024-09-13T09:44:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1526614016 bytes 2024-09-13T09:44:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1534740480 bytes 2024-09-13T09:44:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1536915456 bytes 2024-09-13T09:44:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1584636928 bytes 2024-09-13T09:44:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1670433792 bytes 2024-09-13T09:44:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1707083776 bytes 2024-09-13T09:44:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1738333184 bytes 2024-09-13T09:44:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1739104256 bytes 2024-09-13T09:44:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1777405952 bytes 2024-09-13T09:44:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1810805760 bytes 2024-09-13T09:44:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1868094464 bytes 2024-09-13T09:44:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 1924018176 bytes 2024-09-13T09:44:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 1950723072 bytes 2024-09-13T09:44:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2004440064 bytes 2024-09-13T09:44:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2063451136 bytes 2024-09-13T09:44:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2079029248 bytes 2024-09-13T09:44:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:44:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:44:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:44:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:44:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:44:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_50 2024-09-13T09:44:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2024-09-13T09:44:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:44:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:44:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:44:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:44:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:44:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:44:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:44:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n snapshot ']' 2024-09-13T09:44:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:44:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_50 128M 2024-09-13T09:44:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:44:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:44:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:44:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:44:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:44:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:44:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:44:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 128M ']' 2024-09-13T09:44:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=128M 2024-09-13T09:44:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:44:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_50 2024-09-13T09:44:17.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' image = image ']' 2024-09-13T09:44:17.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2024-09-13T09:44:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:44:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:44:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:44:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:44:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2024-09-13T09:44:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:44:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2024-09-13T09:44:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_50': 2024-09-13T09:44:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:44:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:44:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:44:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 14acd5e15e11 2024-09-13T09:44:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.14acd5e15e11 2024-09-13T09:44:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:44:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:44:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:44:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:44:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:44:16 2024 2024-09-13T09:44:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:44:16 2024 2024-09-13T09:44:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:44:16 2024 2024-09-13T09:44:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:44:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:44:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 6deefad4-7356-4a90-996d-f561ce31e3b2 2024-09-13T09:44:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:44:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n '' ']' 2024-09-13T09:44:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_50 2024-09-13T09:44:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:44:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:44:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:44:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:44:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2024-09-13T09:44:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:44:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:44:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 50299904 bytes 2024-09-13T09:44:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 92570624 bytes 2024-09-13T09:44:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 151074816 bytes 2024-09-13T09:44:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 198792192 bytes 2024-09-13T09:44:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 199419904 bytes 2024-09-13T09:44:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 255336448 bytes 2024-09-13T09:44:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 279889920 bytes 2024-09-13T09:44:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 303392768 bytes 2024-09-13T09:44:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 328946688 bytes 2024-09-13T09:44:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 353379328 bytes 2024-09-13T09:44:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 376356864 bytes 2024-09-13T09:44:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 431500288 bytes 2024-09-13T09:44:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 453474304 bytes 2024-09-13T09:44:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 481763328 bytes 2024-09-13T09:44:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 515904512 bytes 2024-09-13T09:44:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 573001728 bytes 2024-09-13T09:44:19.965 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2024-09-13T09:44:19.963+0000 7f3610499640 -1 reset not still connected to 0x55cdebac5520 2024-09-13T09:44:19.966 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2024-09-13T09:44:19.964+0000 7f3610499640 -1 reset not still connected to 0x55cdec252680 2024-09-13T09:44:19.966 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2024-09-13T09:44:19.964+0000 7f3610499640 -1 reset not still connected to 0x55cdf220dba0 2024-09-13T09:44:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 583622656 bytes 2024-09-13T09:44:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 622164992 bytes 2024-09-13T09:44:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 622769152 bytes 2024-09-13T09:44:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 703679488 bytes 2024-09-13T09:44:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 743706624 bytes 2024-09-13T09:44:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 745538560 bytes 2024-09-13T09:44:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 813593600 bytes 2024-09-13T09:44:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 835514368 bytes 2024-09-13T09:44:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 947957760 bytes 2024-09-13T09:44:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 970377216 bytes 2024-09-13T09:44:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 991475712 bytes 2024-09-13T09:44:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1008269312 bytes 2024-09-13T09:44:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1027225600 bytes 2024-09-13T09:44:21.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1037691904 bytes 2024-09-13T09:44:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1048878080 bytes 2024-09-13T09:44:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1067245568 bytes 2024-09-13T09:44:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1102771200 bytes 2024-09-13T09:44:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1124827136 bytes 2024-09-13T09:44:22.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1183558656 bytes 2024-09-13T09:44:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1208663040 bytes 2024-09-13T09:44:22.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1224164352 bytes 2024-09-13T09:44:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1258491904 bytes 2024-09-13T09:44:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1336584192 bytes 2024-09-13T09:44:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1409302528 bytes 2024-09-13T09:44:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1446771712 bytes 2024-09-13T09:44:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1489756160 bytes 2024-09-13T09:44:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1556512768 bytes 2024-09-13T09:44:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1597521920 bytes 2024-09-13T09:44:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1660977152 bytes 2024-09-13T09:44:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1701138432 bytes 2024-09-13T09:44:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1789403136 bytes 2024-09-13T09:44:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1811139584 bytes 2024-09-13T09:44:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1849884672 bytes 2024-09-13T09:44:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1860734976 bytes 2024-09-13T09:44:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 1883830272 bytes 2024-09-13T09:44:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 1925207040 bytes 2024-09-13T09:44:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 1975307264 bytes 2024-09-13T09:44:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2088682496 bytes 2024-09-13T09:44:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2116843520 bytes 2024-09-13T09:44:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2192468992 bytes 2024-09-13T09:44:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2198544384 bytes 2024-09-13T09:44:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2228524032 bytes 2024-09-13T09:44:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2283165696 bytes 2024-09-13T09:44:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2328156160 bytes 2024-09-13T09:44:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2351262720 bytes 2024-09-13T09:44:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2370742272 bytes 2024-09-13T09:44:26.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2407221248 bytes 2024-09-13T09:44:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2440007680 bytes 2024-09-13T09:44:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2487440384 bytes 2024-09-13T09:44:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2507557888 bytes 2024-09-13T09:44:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2556313600 bytes 2024-09-13T09:44:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2584002560 bytes 2024-09-13T09:44:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2584619008 bytes 2024-09-13T09:44:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2634250240 bytes 2024-09-13T09:44:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2673430528 bytes 2024-09-13T09:44:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2712105984 bytes 2024-09-13T09:44:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2801307648 bytes 2024-09-13T09:44:27.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 2858679296 bytes 2024-09-13T09:44:27.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 2883853312 bytes 2024-09-13T09:44:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 2917230592 bytes 2024-09-13T09:44:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 2996063232 bytes 2024-09-13T09:44:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3061381120 bytes 2024-09-13T09:44:28.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3131301888 bytes 2024-09-13T09:44:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3183394816 bytes 2024-09-13T09:44:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3224487936 bytes 2024-09-13T09:44:28.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3271877632 bytes 2024-09-13T09:44:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3287803904 bytes 2024-09-13T09:44:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3293115392 bytes 2024-09-13T09:44:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3389150208 bytes 2024-09-13T09:44:29.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3450890240 bytes 2024-09-13T09:44:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3476011008 bytes 2024-09-13T09:44:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3536702464 bytes 2024-09-13T09:44:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3560419328 bytes 2024-09-13T09:44:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 3561006080 bytes 2024-09-13T09:44:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 3576117248 bytes 2024-09-13T09:44:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 3592404992 bytes 2024-09-13T09:44:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 3622478848 bytes 2024-09-13T09:44:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:44:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:44:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:44:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:44:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-09-13T09:44:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:44:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:44:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_count=50 2024-09-13T09:44:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:44:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local count 2024-09-13T09:44:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_count=0 2024-09-13T09:44:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:44:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ seq 1 40 2024-09-13T09:44:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in `seq 1 40` 2024-09-13T09:44:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 1 -ne 1 2024-09-13T09:44:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-13T09:44:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep 'images: ' 2024-09-13T09:44:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cut '-d ' -f 2 2024-09-13T09:44:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2024-09-13T09:44:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ count=50 2024-09-13T09:44:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 50 = 50 2024-09-13T09:44:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:44:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-09-13T09:44:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:44:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:44:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:44:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state 2024-09-13T09:44:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ seq 1 40 2024-09-13T09:44:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in `seq 1 40` 2024-09-13T09:44:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 1 -ne 1 2024-09-13T09:44:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-13T09:44:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep 'image health:' 2024-09-13T09:44:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cut '-d ' -f 3 2024-09-13T09:44:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2024-09-13T09:44:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=OK 2024-09-13T09:44:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test OK = ERROR 2024-09-13T09:44:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test OK = OK 2024-09-13T09:44:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:44:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-09-13T09:44:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:44:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:44:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_count=50 2024-09-13T09:44:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:44:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local count 2024-09-13T09:44:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_count=0 2024-09-13T09:44:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:44:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ seq 1 40 2024-09-13T09:44:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in `seq 1 40` 2024-09-13T09:44:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 1 -ne 1 2024-09-13T09:44:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-13T09:44:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep 'images: ' 2024-09-13T09:44:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cut '-d ' -f 2 2024-09-13T09:44:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2024-09-13T09:44:30.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ count=50 2024-09-13T09:44:30.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 50 = 50 2024-09-13T09:44:30.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:44:30.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-09-13T09:44:30.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:44:30.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:44:30.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:44:30.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state 2024-09-13T09:44:30.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ seq 1 40 2024-09-13T09:44:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in `seq 1 40` 2024-09-13T09:44:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 1 -ne 1 2024-09-13T09:44:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-13T09:44:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep 'image health:' 2024-09-13T09:44:30.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cut '-d ' -f 3 2024-09-13T09:44:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2024-09-13T09:44:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=WARNING 2024-09-13T09:44:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test WARNING = ERROR 2024-09-13T09:44:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test WARNING = OK 2024-09-13T09:44:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in `seq 1 40` 2024-09-13T09:44:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 2 -ne 1 2024-09-13T09:44:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 30 2024-09-13T09:44:54.526 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-13T09:44:54.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-13T09:44:54.527 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-13T09:44:54.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T09:44:54.526+0000 7f521e3ce640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 47204) UID: 1000 2024-09-13T09:45:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-13T09:45:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep 'image health:' 2024-09-13T09:45:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cut '-d ' -f 3 2024-09-13T09:45:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2024-09-13T09:45:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=OK 2024-09-13T09:45:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test OK = ERROR 2024-09-13T09:45:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test OK = OK 2024-09-13T09:45:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:45:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ testlog 'TEST: compare many images' 2024-09-13T09:45:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee -a /tmp/tmp.V7YewHEkvl/rbd-mirror.test.log 2024-09-13T09:45:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ date '+%F %T' 2024-09-13T09:45:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo 2024-09-13 09:45:00 TEST: compare many images 2024-09-13T09:45:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:2024-09-13 09:45:00 TEST: compare many images 2024-09-13T09:45:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ seq 1 50 2024-09-13T09:45:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:45:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_1 2024-09-13T09:45:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_1 snap 2024-09-13T09:45:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:45:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:45:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:45:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-09-13T09:45:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:45:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-09-13T09:45:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:45:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:45:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:45:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:45:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:45:02.528 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-13T09:45:02.528 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:45:02.529 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-13T09:45:02.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-13T09:45:02.529 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-13T09:45:02.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-13T09:45:02.533 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2024-09-13T09:45:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-09-13T09:45:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:45:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:45:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:45:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:45:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-09-13T09:45:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:45:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:45:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:45:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:45:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:45:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:45:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:45:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:45:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:45:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-09-13T09:45:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:45:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:45:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:45:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:45:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:45:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:45:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:45:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:45:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:45:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:45:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:45:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:45:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:45:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:45:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:45:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:45:02.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:02.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:02.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:02.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-09-13T09:45:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-09-13T09:45:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:45:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2024-09-13T09:45:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:45:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:45:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:45:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:45:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:45:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:45:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:45:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:45:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:45:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:45:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:45:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:45:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:45:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:45:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:45:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:45:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2024-09-13T09:45:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:45:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:45:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:45:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:45:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:45:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:45:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:45:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:45:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-09-13T09:45:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:45:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:45:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:45:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:45:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2024-09-13T09:45:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:45:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:45:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_1.status 2024-09-13T09:45:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_1.status 2024-09-13T09:45:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:45:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_1.status 2024-09-13T09:45:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_1.status 2024-09-13T09:45:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_1.status 2024-09-13T09:45:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:45:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_1.status 2024-09-13T09:45:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2024-09-13T09:45:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:45:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2024-09-13T09:45:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 119 2024-09-13T09:45:03.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_1.status 2024-09-13T09:45:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:45:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_1.status 2024-09-13T09:45:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2024-09-13T09:45:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:45:04.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=119 2024-09-13T09:45:04.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:45:04.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:45:04.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:45:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_1.status 2024-09-13T09:45:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:45:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_1.status 2024-09-13T09:45:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-13T09:45:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:45:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=68 2024-09-13T09:45:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 119 = 68 2024-09-13T09:45:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:45:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:45:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_1.status 2024-09-13T09:45:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:45:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_1.status 2024-09-13T09:45:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-13T09:45:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:45:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:45:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:45:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:45:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:45:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:45:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 119 = '' 2024-09-13T09:45:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:45:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:45:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_1.status 2024-09-13T09:45:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:45:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_1.status 2024-09-13T09:45:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-13T09:45:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:45:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:45:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:45:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:45:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:45:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:45:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 119 = '' 2024-09-13T09:45:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:45:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:45:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_1.status 2024-09-13T09:45:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:45:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_1.status 2024-09-13T09:45:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-13T09:45:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:45:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=119 2024-09-13T09:45:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 119 = 119 2024-09-13T09:45:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:45:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-09-13T09:45:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:45:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:45:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:45:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:45:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:45:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-09-13T09:45:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_1': 2024-09-13T09:45:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:45:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:45:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:45:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c56af2347a 2024-09-13T09:45:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c56af2347a 2024-09-13T09:45:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:45:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:45:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:45:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:45:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:35:53 2024 2024-09-13T09:45:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:35:53 2024 2024-09-13T09:45:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:35:53 2024 2024-09-13T09:45:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:45:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:45:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:45:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 6c1c05c0-9a83-4822-98dd-9fa3a496df28 2024-09-13T09:45:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:45:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:45:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_1 snap 2024-09-13T09:45:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:45:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:45:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:45:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_1.export 2024-09-13T09:45:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_1.export 2024-09-13T09:45:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_1.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_1.export 2024-09-13T09:45:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_1.export 2024-09-13T09:45:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:45:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_1.export 2024-09-13T09:45:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:45:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_1.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_1.export 2024-09-13T09:45:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_1.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_1.export 2024-09-13T09:45:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:45:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:45:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_2 2024-09-13T09:45:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_2 snap 2024-09-13T09:45:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:45:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:45:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:45:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-09-13T09:45:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:45:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-09-13T09:45:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:45:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:45:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:45:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:45:09.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:45:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-09-13T09:45:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:45:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:45:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:45:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:45:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-09-13T09:45:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:45:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:45:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:45:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:45:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:45:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:45:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:45:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:45:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:45:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-09-13T09:45:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:45:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:45:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:45:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:45:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:45:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:45:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:45:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:45:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:45:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:45:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:45:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:45:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:45:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:45:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:45:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:45:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-09-13T09:45:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 2 closest matches: 2024-09-13T09:45:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_21 2024-09-13T09:45:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_23 2024-09-13T09:45:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2024-09-13T09:45:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:45:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:45:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:45:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:45:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:45:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:45:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:45:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:45:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:45:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:45:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:45:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:45:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:45:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:45:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:45:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:45:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2024-09-13T09:45:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-09-13T09:45:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:45:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2024-09-13T09:45:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:45:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:45:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:45:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:45:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:45:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:45:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:45:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:45:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:45:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:45:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:45:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:45:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:45:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:45:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:45:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:45:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:45:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:45:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:45:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2024-09-13T09:45:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 1 closest matches: 2024-09-13T09:45:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_20 2024-09-13T09:45:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2024-09-13T09:45:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:45:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:45:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:45:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:45:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:45:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:45:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:45:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:45:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:45:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:45:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:45:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:45:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:45:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:45:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:45:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:45:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:45:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:45:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:45:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2024-09-13T09:45:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 3 closest matches: 2024-09-13T09:45:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_24 2024-09-13T09:45:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_25 2024-09-13T09:45:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_29 2024-09-13T09:45:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:45:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 1 2024-09-13T09:45:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:45:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-09-13T09:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:45:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:45:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:45:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:45:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:45:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:45:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:45:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:45:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:45:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:45:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:45:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:45:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-09-13T09:45:12.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 2 closest matches: 2024-09-13T09:45:12.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_21 2024-09-13T09:45:12.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_23 2024-09-13T09:45:12.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2024-09-13T09:45:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:45:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:45:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:45:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:45:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:45:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:45:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:45:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:45:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:45:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:45:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2024-09-13T09:45:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-09-13T09:45:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:45:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2024-09-13T09:45:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:45:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:45:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:45:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:45:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:45:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:45:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:45:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:45:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:45:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:45:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:45:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:45:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:45:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:45:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:45:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:45:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:45:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:45:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:45:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2024-09-13T09:45:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 1 closest matches: 2024-09-13T09:45:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_20 2024-09-13T09:45:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:12.696 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2024-09-13T09:45:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:45:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:45:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:45:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:45:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:45:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:45:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:45:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:45:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:45:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:45:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:45:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:45:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:45:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:12.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:45:12.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:45:12.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:45:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:45:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:45:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:45:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2024-09-13T09:45:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 3 closest matches: 2024-09-13T09:45:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_24 2024-09-13T09:45:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_25 2024-09-13T09:45:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_29 2024-09-13T09:45:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:45:12.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 2 2024-09-13T09:45:14.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:45:14.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:45:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-09-13T09:45:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:45:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:45:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:45:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:45:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:45:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:45:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:45:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:45:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:45:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:45:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:45:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:45:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:45:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:45:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:45:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:45:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-09-13T09:45:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 2 closest matches: 2024-09-13T09:45:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_21 2024-09-13T09:45:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_23 2024-09-13T09:45:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2024-09-13T09:45:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:45:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:45:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:45:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:45:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:45:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:45:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:45:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:45:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:45:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:45:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:45:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:45:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:45:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:45:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:45:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:45:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2024-09-13T09:45:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-09-13T09:45:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:45:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2024-09-13T09:45:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:45:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:45:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:45:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:45:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:45:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:45:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:45:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:45:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:45:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:45:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:45:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:45:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:45:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:45:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:45:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:45:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:45:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:45:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:45:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2024-09-13T09:45:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 1 closest matches: 2024-09-13T09:45:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_20 2024-09-13T09:45:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2024-09-13T09:45:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:45:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:45:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:45:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:45:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:45:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:45:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:45:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:45:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:45:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:45:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:45:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:45:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:45:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:45:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:45:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:45:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:45:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:45:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:45:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2024-09-13T09:45:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 3 closest matches: 2024-09-13T09:45:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_24 2024-09-13T09:45:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_25 2024-09-13T09:45:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_29 2024-09-13T09:45:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:45:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 4 2024-09-13T09:45:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:45:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:45:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-09-13T09:45:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:45:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:45:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:45:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:45:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:45:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:45:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:45:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:45:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:45:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:45:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:45:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:45:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:45:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:45:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:45:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:45:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-09-13T09:45:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 2 closest matches: 2024-09-13T09:45:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_21 2024-09-13T09:45:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_23 2024-09-13T09:45:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2024-09-13T09:45:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:45:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:45:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:45:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:45:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:45:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:45:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:45:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:45:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:45:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:45:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:45:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:45:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:45:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:45:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:45:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:45:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2024-09-13T09:45:19.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-09-13T09:45:19.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:45:19.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2024-09-13T09:45:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:45:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:45:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:45:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:45:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:45:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:45:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:45:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:45:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:45:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:45:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:45:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:45:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:45:19.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:19.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:19.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:45:19.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:45:19.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:45:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:45:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:45:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:45:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2024-09-13T09:45:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 1 closest matches: 2024-09-13T09:45:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_20 2024-09-13T09:45:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2024-09-13T09:45:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:45:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:45:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:45:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:45:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:45:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:45:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:45:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:45:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:45:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:45:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:45:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:45:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:45:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:45:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:45:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:45:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:45:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:45:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:45:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2024-09-13T09:45:19.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 3 closest matches: 2024-09-13T09:45:19.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_24 2024-09-13T09:45:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_25 2024-09-13T09:45:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_29 2024-09-13T09:45:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:45:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 8 2024-09-13T09:45:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:45:27.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:45:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-09-13T09:45:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:45:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:45:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:45:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:45:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:45:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:45:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:45:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:45:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:45:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:45:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:45:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:45:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:45:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:45:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:45:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:45:27.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:27.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:27.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:27.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-09-13T09:45:27.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:45:27.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:45:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:45:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:45:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:45:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:45:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:45:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:45:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-09-13T09:45:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:45:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:45:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:45:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:45:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2024-09-13T09:45:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:45:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:45:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_2.status 2024-09-13T09:45:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_2.status 2024-09-13T09:45:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:45:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_2.status 2024-09-13T09:45:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_2.status 2024-09-13T09:45:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_2.status 2024-09-13T09:45:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:45:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_2.status 2024-09-13T09:45:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2024-09-13T09:45:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:45:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2024-09-13T09:45:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 121 2024-09-13T09:45:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_2.status 2024-09-13T09:45:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:45:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_2.status 2024-09-13T09:45:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2024-09-13T09:45:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:45:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=121 2024-09-13T09:45:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:45:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:45:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:45:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_2.status 2024-09-13T09:45:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:45:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_2.status 2024-09-13T09:45:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-13T09:45:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:45:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=69 2024-09-13T09:45:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 121 = 69 2024-09-13T09:45:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:45:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:45:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_2.status 2024-09-13T09:45:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:45:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_2.status 2024-09-13T09:45:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-13T09:45:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:45:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:45:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:45:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:45:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:45:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:45:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 121 = '' 2024-09-13T09:45:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:45:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:45:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_2.status 2024-09-13T09:45:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:45:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_2.status 2024-09-13T09:45:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-13T09:45:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:45:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:45:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:45:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:45:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:45:29.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:45:29.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 121 = '' 2024-09-13T09:45:29.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:45:29.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:45:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_2.status 2024-09-13T09:45:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:45:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_2.status 2024-09-13T09:45:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-13T09:45:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:45:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=121 2024-09-13T09:45:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 121 = 121 2024-09-13T09:45:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:45:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-09-13T09:45:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:45:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:45:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:45:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:45:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:45:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-09-13T09:45:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_2': 2024-09-13T09:45:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:45:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:45:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:45:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 114ea0b62a41 2024-09-13T09:45:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.114ea0b62a41 2024-09-13T09:45:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:45:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:45:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:45:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:45:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:36:07 2024 2024-09-13T09:45:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:36:07 2024 2024-09-13T09:45:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:36:07 2024 2024-09-13T09:45:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:45:32.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:45:32.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:45:32.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: c4b5aac5-000e-4a00-bd35-23bce99cad5f 2024-09-13T09:45:32.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:45:32.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:45:32.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_2 snap 2024-09-13T09:45:32.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:32.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:45:32.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:45:32.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:45:32.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_2.export 2024-09-13T09:45:32.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_2.export 2024-09-13T09:45:32.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_2.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_2.export 2024-09-13T09:45:32.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_2.export 2024-09-13T09:45:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:45:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_2.export 2024-09-13T09:45:32.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:45:32.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_2.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_2.export 2024-09-13T09:45:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_2.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_2.export 2024-09-13T09:45:32.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:45:32.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:45:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_3 2024-09-13T09:45:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_3 snap 2024-09-13T09:45:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:45:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:45:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:45:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-09-13T09:45:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:45:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-09-13T09:45:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:45:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:45:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:45:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:45:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:45:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-09-13T09:45:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:45:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:45:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:45:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:45:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-09-13T09:45:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:45:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:45:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:45:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:45:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:45:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:45:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:45:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:45:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:45:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-09-13T09:45:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:45:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:45:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:45:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:45:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:45:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:45:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:45:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:45:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:45:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:45:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:45:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:45:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:45:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:45:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:45:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:45:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-09-13T09:45:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2024-09-13T09:45:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:45:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2024-09-13T09:45:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:45:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:45:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:45:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:45:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:45:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:45:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:45:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:45:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:45:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:45:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:45:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:45:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:45:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:45:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:45:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:45:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2024-09-13T09:45:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:45:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:45:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:45:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:45:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:45:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:45:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:45:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:45:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-09-13T09:45:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:45:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:45:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:45:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:45:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2024-09-13T09:45:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:45:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:45:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_3.status 2024-09-13T09:45:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_3.status 2024-09-13T09:45:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:45:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_3.status 2024-09-13T09:45:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_3.status 2024-09-13T09:45:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_3.status 2024-09-13T09:45:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:45:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_3.status 2024-09-13T09:45:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2024-09-13T09:45:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:45:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2024-09-13T09:45:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 123 2024-09-13T09:45:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_3.status 2024-09-13T09:45:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:45:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_3.status 2024-09-13T09:45:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2024-09-13T09:45:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:45:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=123 2024-09-13T09:45:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:45:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:45:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:45:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_3.status 2024-09-13T09:45:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:45:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_3.status 2024-09-13T09:45:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-13T09:45:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:45:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=70 2024-09-13T09:45:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 123 = 70 2024-09-13T09:45:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:45:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:45:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_3.status 2024-09-13T09:45:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:45:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_3.status 2024-09-13T09:45:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-13T09:45:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:45:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:45:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:45:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:45:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:45:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:45:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 123 = '' 2024-09-13T09:45:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:45:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:45:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_3.status 2024-09-13T09:45:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:45:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_3.status 2024-09-13T09:45:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-13T09:45:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:45:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:45:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:45:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:45:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:45:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:45:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 123 = '' 2024-09-13T09:45:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:45:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:45:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_3.status 2024-09-13T09:45:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:45:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_3.status 2024-09-13T09:45:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-13T09:45:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:45:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=123 2024-09-13T09:45:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 123 = 123 2024-09-13T09:45:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:45:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-09-13T09:45:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:45:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:45:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:45:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:45:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:45:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-09-13T09:45:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_3': 2024-09-13T09:45:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:45:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 11ce8657d7f8 2024-09-13T09:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.11ce8657d7f8 2024-09-13T09:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:36:16 2024 2024-09-13T09:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:36:16 2024 2024-09-13T09:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:36:16 2024 2024-09-13T09:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: c99ca9ae-d4dc-48db-a446-dd6058760fec 2024-09-13T09:45:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:45:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:45:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_3 snap 2024-09-13T09:45:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:45:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:45:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:45:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_3.export 2024-09-13T09:45:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_3.export 2024-09-13T09:45:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_3.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_3.export 2024-09-13T09:45:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_3.export 2024-09-13T09:45:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:45:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_3.export 2024-09-13T09:45:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:45:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_3.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_3.export 2024-09-13T09:45:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_3.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_3.export 2024-09-13T09:45:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:45:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:45:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_4 2024-09-13T09:45:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_4 snap 2024-09-13T09:45:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:45:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:45:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:45:41.916 INFO:tasks.ceph.cluster1.osd.0.smithi122.stderr:2024-09-13T09:45:41.914+0000 7ff198c84640 -1 reset not still connected to 0x560ff289af70 2024-09-13T09:45:41.917 INFO:tasks.ceph.cluster1.osd.0.smithi122.stderr:2024-09-13T09:45:41.914+0000 7ff198c84640 -1 reset not still connected to 0x560ff289bba0 2024-09-13T09:45:41.917 INFO:tasks.ceph.cluster1.osd.0.smithi122.stderr:2024-09-13T09:45:41.914+0000 7ff198c84640 -1 reset not still connected to 0x560ff3a26820 2024-09-13T09:45:41.917 INFO:tasks.ceph.cluster1.osd.0.smithi122.stderr:2024-09-13T09:45:41.914+0000 7ff198c84640 -1 reset not still connected to 0x560ff3b65520 2024-09-13T09:45:41.917 INFO:tasks.ceph.cluster1.osd.0.smithi122.stderr:2024-09-13T09:45:41.914+0000 7ff198c84640 -1 reset not still connected to 0x560ff3b65ad0 2024-09-13T09:45:41.917 INFO:tasks.ceph.cluster1.osd.0.smithi122.stderr:2024-09-13T09:45:41.914+0000 7ff198c84640 -1 reset not still connected to 0x560ff46381a0 2024-09-13T09:45:41.917 INFO:tasks.ceph.cluster1.osd.0.smithi122.stderr:2024-09-13T09:45:41.914+0000 7ff198c84640 -1 reset not still connected to 0x560ff4c291e0 2024-09-13T09:45:41.917 INFO:tasks.ceph.cluster1.osd.0.smithi122.stderr:2024-09-13T09:45:41.915+0000 7ff198c84640 -1 reset not still connected to 0x560ff5582a90 2024-09-13T09:45:41.917 INFO:tasks.ceph.cluster1.osd.0.smithi122.stderr:2024-09-13T09:45:41.915+0000 7ff198c84640 -1 reset not still connected to 0x560ff5583450 2024-09-13T09:45:41.917 INFO:tasks.ceph.cluster1.osd.0.smithi122.stderr:2024-09-13T09:45:41.915+0000 7ff198c84640 -1 reset not still connected to 0x560ff5583930 2024-09-13T09:45:41.917 INFO:tasks.ceph.cluster1.osd.0.smithi122.stderr:2024-09-13T09:45:41.915+0000 7ff198c84640 -1 reset not still connected to 0x560ff5c4c1a0 2024-09-13T09:45:41.917 INFO:tasks.ceph.cluster1.osd.0.smithi122.stderr:2024-09-13T09:45:41.915+0000 7ff198c84640 -1 reset not still connected to 0x560ff622edd0 2024-09-13T09:45:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-09-13T09:45:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:45:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-09-13T09:45:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:45:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:45:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:45:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:45:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:45:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-09-13T09:45:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:45:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:45:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:45:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:45:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-09-13T09:45:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:45:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:45:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:45:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:45:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:45:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:45:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:45:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:45:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:45:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-09-13T09:45:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:45:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:45:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:45:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:45:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:45:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:45:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:45:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:45:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:45:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:45:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:45:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:45:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:45:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:45:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:45:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:45:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-09-13T09:45:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 1 closest matches: 2024-09-13T09:45:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_40 2024-09-13T09:45:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2024-09-13T09:45:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:45:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:45:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:45:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:45:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:45:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:45:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:45:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:45:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:45:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:45:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:45:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:45:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:45:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:45:44.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:45:44.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:45:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2024-09-13T09:45:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 3 closest matches: 2024-09-13T09:45:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_46 2024-09-13T09:45:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_47 2024-09-13T09:45:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_48 2024-09-13T09:45:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:44.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2024-09-13T09:45:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:45:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:45:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:45:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:45:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:45:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:45:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:45:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:45:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:45:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:45:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:45:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:45:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:45:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:45:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:45:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:45:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:45:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:45:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:45:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2024-09-13T09:45:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:no valid command found; 2 closest matches: 2024-09-13T09:45:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_42 2024-09-13T09:45:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status mirror/image_43 2024-09-13T09:45:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2024-09-13T09:45:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:45:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:45:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:45:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:45:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:45:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:45:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:45:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:45:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:45:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:45:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:45:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:45:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:45:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:45:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:45:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:45:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:45:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:45:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:45:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2024-09-13T09:45:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:45:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:45:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:45:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:45:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:45:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:45:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:45:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:45:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-09-13T09:45:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:45:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:45:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:45:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:45:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2024-09-13T09:45:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:45:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:45:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_4.status 2024-09-13T09:45:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_4.status 2024-09-13T09:45:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:45:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_4.status 2024-09-13T09:45:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_4.status 2024-09-13T09:45:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_4.status 2024-09-13T09:45:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:45:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_4.status 2024-09-13T09:45:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2024-09-13T09:45:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:45:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2024-09-13T09:45:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 125 2024-09-13T09:45:45.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_4.status 2024-09-13T09:45:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:45:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_4.status 2024-09-13T09:45:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2024-09-13T09:45:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:45:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=125 2024-09-13T09:45:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:45:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:45:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:45:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_4.status 2024-09-13T09:45:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:45:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_4.status 2024-09-13T09:45:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-13T09:45:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:45:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=71 2024-09-13T09:45:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 125 = 71 2024-09-13T09:45:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:45:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:45:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_4.status 2024-09-13T09:45:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:45:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_4.status 2024-09-13T09:45:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-13T09:45:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:45:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:45:45.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:45:45.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:45:45.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:45:45.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:45:45.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 125 = '' 2024-09-13T09:45:45.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:45:45.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:45:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_4.status 2024-09-13T09:45:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:45:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_4.status 2024-09-13T09:45:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-13T09:45:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:45:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:45:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:45:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:45:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:45:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:45:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 125 = '' 2024-09-13T09:45:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:45:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:45:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_4.status 2024-09-13T09:45:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:45:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_4.status 2024-09-13T09:45:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-13T09:45:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:45:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=125 2024-09-13T09:45:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 125 = 125 2024-09-13T09:45:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:45:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-09-13T09:45:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:45:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:45:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:45:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:45:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:45:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-09-13T09:45:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_4': 2024-09-13T09:45:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:45:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:45:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:45:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 11f99c487973 2024-09-13T09:45:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.11f99c487973 2024-09-13T09:45:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:45:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:45:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:45:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:45:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:36:26 2024 2024-09-13T09:45:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:36:26 2024 2024-09-13T09:45:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:36:26 2024 2024-09-13T09:45:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:45:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:45:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:45:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 46dd7eaa-6bb3-4d95-bf94-ddb77bae7a9e 2024-09-13T09:45:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:45:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:45:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_4 snap 2024-09-13T09:45:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:45:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:45:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:45:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_4.export 2024-09-13T09:45:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_4.export 2024-09-13T09:45:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_4.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_4.export 2024-09-13T09:45:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_4.export 2024-09-13T09:45:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:45:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_4.export 2024-09-13T09:45:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:45:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_4.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_4.export 2024-09-13T09:45:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_4.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_4.export 2024-09-13T09:45:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:45:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:45:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_5 2024-09-13T09:45:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_5 snap 2024-09-13T09:45:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:45:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:45:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:45:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-09-13T09:45:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:45:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-09-13T09:45:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:45:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:45:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:45:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:45:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:45:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-09-13T09:45:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:45:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:45:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:45:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:45:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-09-13T09:45:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:45:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:45:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:45:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:45:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:45:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:45:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:45:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:45:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:45:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-09-13T09:45:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:45:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:45:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:45:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:45:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:45:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:45:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:45:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:45:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:45:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:45:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:45:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:45:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:45:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:45:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:45:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:45:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:45:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-09-13T09:45:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-13T09:45:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:45:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:45:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:45:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-09-13T09:45:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:45:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:45:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:45:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:45:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:45:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:45:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:45:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:45:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:45:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:45:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:45:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:45:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:45:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:45:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:45:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:45:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:45:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:45:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:45:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:45:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:45:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:45:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:45:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:45:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:45:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:45:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:45:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:45:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:45:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:45:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-09-13T09:45:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:45:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:45:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:45:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:45:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:45:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:45:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:45:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:45:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-09-13T09:45:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:45:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:45:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:45:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:45:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2024-09-13T09:45:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:45:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:45:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_5.status 2024-09-13T09:45:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_5.status 2024-09-13T09:45:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:45:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_5.status 2024-09-13T09:45:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_5.status 2024-09-13T09:45:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_5.status 2024-09-13T09:45:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:45:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_5.status 2024-09-13T09:45:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2024-09-13T09:45:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:45:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2024-09-13T09:45:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 127 2024-09-13T09:45:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_5.status 2024-09-13T09:45:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:45:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_5.status 2024-09-13T09:45:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2024-09-13T09:45:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:45:53.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=127 2024-09-13T09:45:53.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:45:53.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:45:53.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:45:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_5.status 2024-09-13T09:45:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:45:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_5.status 2024-09-13T09:45:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-13T09:45:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:45:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=72 2024-09-13T09:45:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 127 = 72 2024-09-13T09:45:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:45:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:45:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_5.status 2024-09-13T09:45:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:45:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_5.status 2024-09-13T09:45:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-13T09:45:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:45:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:45:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:45:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:45:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:45:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:45:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 127 = '' 2024-09-13T09:45:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:45:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:45:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_5.status 2024-09-13T09:45:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:45:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_5.status 2024-09-13T09:45:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-13T09:45:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:45:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:45:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:45:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:45:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:45:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:45:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 127 = '' 2024-09-13T09:45:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:45:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:45:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_5.status 2024-09-13T09:45:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:45:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_5.status 2024-09-13T09:45:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-13T09:45:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:45:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:45:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=127 2024-09-13T09:45:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 127 = 127 2024-09-13T09:45:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:45:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-09-13T09:45:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:45:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:45:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:45:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:45:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:45:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:45:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-09-13T09:45:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_5': 2024-09-13T09:45:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:45:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:45:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:45:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c5d0d91486 2024-09-13T09:45:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c5d0d91486 2024-09-13T09:45:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:45:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:45:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:45:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:45:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:36:38 2024 2024-09-13T09:45:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:36:38 2024 2024-09-13T09:45:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:36:38 2024 2024-09-13T09:45:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:45:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:45:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:45:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 225d4f13-dab0-4960-904b-d2c54fac25f4 2024-09-13T09:45:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:45:57.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:45:57.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_5 snap 2024-09-13T09:45:57.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:45:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:45:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:45:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_5.export 2024-09-13T09:45:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_5.export 2024-09-13T09:45:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_5.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_5.export 2024-09-13T09:45:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_5.export 2024-09-13T09:45:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:45:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_5.export 2024-09-13T09:45:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:45:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_5.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_5.export 2024-09-13T09:45:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_5.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_5.export 2024-09-13T09:45:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:45:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:45:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_6 2024-09-13T09:45:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_6 snap 2024-09-13T09:45:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:45:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:45:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:45:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:45:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:45:58.424 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6114955499404657) > weight (0.25) 2024-09-13T09:45:58.424 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.598619356444817) > weight (0.25) 2024-09-13T09:45:58.424 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-13T09:45:58.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-13T09:45:58.425 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-13T09:45:58.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:45:58.424+0000 7f2aadc53640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 48149) UID: 1000 2024-09-13T09:46:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-09-13T09:46:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:46:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-09-13T09:46:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:46:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:46:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:46:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:46:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:46:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-09-13T09:46:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:46:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:46:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:46:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:46:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-09-13T09:46:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:46:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:46:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:46:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:46:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:46:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:46:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:46:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:46:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:46:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:46:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-09-13T09:46:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:46:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:46:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:46:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:46:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:46:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:46:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:46:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:46:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:46:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:46:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:46:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:46:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:46:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:46:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:46:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:46:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:46:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:46:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:46:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:46:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:46:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:46:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:46:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:46:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:46:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:46:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:46:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-09-13T09:46:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:46:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:46:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:46:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:46:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:46:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:46:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:46:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-09-13T09:46:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:46:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:46:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:46:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:46:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2024-09-13T09:46:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:46:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:46:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_6.status 2024-09-13T09:46:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_6.status 2024-09-13T09:46:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:46:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_6.status 2024-09-13T09:46:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_6.status 2024-09-13T09:46:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_6.status 2024-09-13T09:46:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:46:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_6.status 2024-09-13T09:46:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2024-09-13T09:46:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:46:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2024-09-13T09:46:02.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 129 2024-09-13T09:46:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_6.status 2024-09-13T09:46:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:46:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_6.status 2024-09-13T09:46:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2024-09-13T09:46:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:46:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=129 2024-09-13T09:46:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:46:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:46:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_6.status 2024-09-13T09:46:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:46:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_6.status 2024-09-13T09:46:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-13T09:46:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=73 2024-09-13T09:46:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 129 = 73 2024-09-13T09:46:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:46:02.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_6.status 2024-09-13T09:46:02.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:02.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:02.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:46:02.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_6.status 2024-09-13T09:46:02.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-13T09:46:02.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:46:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:46:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:46:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:46:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:46:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 129 = '' 2024-09-13T09:46:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:46:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_6.status 2024-09-13T09:46:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:46:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_6.status 2024-09-13T09:46:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-13T09:46:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:03.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:46:03.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:46:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:46:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:46:03.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:46:03.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 129 = '' 2024-09-13T09:46:03.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:03.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:46:04.426 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-13T09:46:04.426 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:46:04.427 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-13T09:46:04.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-13T09:46:04.427 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-13T09:46:04.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-13T09:46:04.430 INFO:tasks.rbd_mirror_thrash:waiting for 79 secs before thrashing 2024-09-13T09:46:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_6.status 2024-09-13T09:46:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:46:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_6.status 2024-09-13T09:46:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-13T09:46:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=129 2024-09-13T09:46:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 129 = 129 2024-09-13T09:46:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-09-13T09:46:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:46:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:46:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:46:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:46:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-09-13T09:46:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_6': 2024-09-13T09:46:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:46:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:46:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:46:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 114e148fed5 2024-09-13T09:46:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.114e148fed5 2024-09-13T09:46:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:46:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:46:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:46:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:46:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:36:47 2024 2024-09-13T09:46:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:36:47 2024 2024-09-13T09:46:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:36:47 2024 2024-09-13T09:46:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:46:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:46:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:46:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 140af9b2-8074-4c94-96e5-2095bff8e54c 2024-09-13T09:46:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:46:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_6 snap 2024-09-13T09:46:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:46:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:46:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_6.export 2024-09-13T09:46:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_6.export 2024-09-13T09:46:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_6.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_6.export 2024-09-13T09:46:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_6.export 2024-09-13T09:46:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:46:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_6.export 2024-09-13T09:46:07.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:46:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_6.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_6.export 2024-09-13T09:46:07.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_6.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_6.export 2024-09-13T09:46:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:46:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_7 2024-09-13T09:46:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_7 snap 2024-09-13T09:46:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:46:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:46:09.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-09-13T09:46:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:46:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-09-13T09:46:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:46:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:46:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:46:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:46:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:46:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-09-13T09:46:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:46:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:46:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:46:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:46:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-09-13T09:46:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:46:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:46:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:46:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:46:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:46:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:46:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:46:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:46:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:46:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:46:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-09-13T09:46:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:46:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:46:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:46:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:46:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:46:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:46:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:46:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:46:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:46:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:46:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:46:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:46:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:46:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:46:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:46:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:46:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:46:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:46:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:46:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:46:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:46:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:46:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:46:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:46:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:46:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:46:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:46:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-09-13T09:46:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-09-13T09:46:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:46:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:46:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:46:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2024-09-13T09:46:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:46:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:46:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:46:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:46:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:46:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:46:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:46:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:46:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:46:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:46:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:46:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:46:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:46:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:46:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:46:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:46:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:46:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:46:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:46:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:46:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:46:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:46:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:46:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:46:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:46:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:46:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:46:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:46:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:46:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:46:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2024-09-13T09:46:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:46:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:46:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:46:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:46:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:46:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:46:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:46:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-09-13T09:46:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:46:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:46:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:46:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:46:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2024-09-13T09:46:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:46:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:46:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_7.status 2024-09-13T09:46:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_7.status 2024-09-13T09:46:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:46:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_7.status 2024-09-13T09:46:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_7.status 2024-09-13T09:46:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_7.status 2024-09-13T09:46:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:46:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_7.status 2024-09-13T09:46:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2024-09-13T09:46:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:46:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2024-09-13T09:46:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 131 2024-09-13T09:46:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_7.status 2024-09-13T09:46:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:46:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_7.status 2024-09-13T09:46:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2024-09-13T09:46:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:11.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:46:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=131 2024-09-13T09:46:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:46:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:46:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_7.status 2024-09-13T09:46:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:46:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_7.status 2024-09-13T09:46:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-13T09:46:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:11.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=74 2024-09-13T09:46:11.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 131 = 74 2024-09-13T09:46:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:46:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_7.status 2024-09-13T09:46:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:46:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_7.status 2024-09-13T09:46:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-13T09:46:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:46:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:46:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:46:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:46:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:46:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 131 = '' 2024-09-13T09:46:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:46:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_7.status 2024-09-13T09:46:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:46:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_7.status 2024-09-13T09:46:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-13T09:46:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:46:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:46:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:46:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:46:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:46:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 131 = '' 2024-09-13T09:46:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:46:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_7.status 2024-09-13T09:46:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:46:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_7.status 2024-09-13T09:46:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-13T09:46:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=131 2024-09-13T09:46:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 131 = 131 2024-09-13T09:46:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-09-13T09:46:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:46:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:46:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:46:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:46:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-09-13T09:46:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_7': 2024-09-13T09:46:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:46:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:46:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:46:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c570489888 2024-09-13T09:46:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c570489888 2024-09-13T09:46:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:46:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:46:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:46:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:46:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:37:24 2024 2024-09-13T09:46:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:37:24 2024 2024-09-13T09:46:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:37:24 2024 2024-09-13T09:46:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:46:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:46:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:46:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 05fdfd86-7dcf-47af-bad1-344a4f52a3bd 2024-09-13T09:46:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:46:15.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_7 snap 2024-09-13T09:46:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:46:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:46:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_7.export 2024-09-13T09:46:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_7.export 2024-09-13T09:46:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_7.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_7.export 2024-09-13T09:46:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_7.export 2024-09-13T09:46:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:46:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_7.export 2024-09-13T09:46:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:46:16.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_7.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_7.export 2024-09-13T09:46:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_7.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_7.export 2024-09-13T09:46:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:46:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_8 2024-09-13T09:46:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_8 snap 2024-09-13T09:46:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:46:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:46:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-09-13T09:46:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:46:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-09-13T09:46:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:46:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:46:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:46:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:46:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:46:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-09-13T09:46:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:46:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:46:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:46:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:46:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-09-13T09:46:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:46:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:46:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:46:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:46:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:46:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:46:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:46:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:46:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:46:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:46:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-09-13T09:46:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:46:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:46:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:46:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:46:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:46:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:46:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:46:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:46:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:46:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:46:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:46:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:46:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:46:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:46:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:46:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:46:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:46:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:46:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:46:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:46:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:46:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:46:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:46:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:46:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:46:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:46:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:46:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-09-13T09:46:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-09-13T09:46:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:46:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:46:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:46:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2024-09-13T09:46:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:46:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:46:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:46:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:46:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:46:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:46:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:46:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:46:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:46:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:46:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:46:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:46:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:46:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:46:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:46:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:46:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:46:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:46:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:46:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:46:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:46:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:46:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:46:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:46:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:46:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:46:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:46:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:46:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:46:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:46:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2024-09-13T09:46:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:46:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:46:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:46:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:46:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:46:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:46:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:46:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-09-13T09:46:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:46:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:46:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:46:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:46:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2024-09-13T09:46:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:46:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:46:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_8.status 2024-09-13T09:46:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_8.status 2024-09-13T09:46:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:46:19.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_8.status 2024-09-13T09:46:19.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_8.status 2024-09-13T09:46:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_8.status 2024-09-13T09:46:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:46:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_8.status 2024-09-13T09:46:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2024-09-13T09:46:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:46:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2024-09-13T09:46:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 133 2024-09-13T09:46:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_8.status 2024-09-13T09:46:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:46:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_8.status 2024-09-13T09:46:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2024-09-13T09:46:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:46:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=133 2024-09-13T09:46:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:46:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:46:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_8.status 2024-09-13T09:46:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:46:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_8.status 2024-09-13T09:46:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-13T09:46:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=75 2024-09-13T09:46:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 133 = 75 2024-09-13T09:46:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:46:20.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_8.status 2024-09-13T09:46:20.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:20.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:20.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:46:20.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_8.status 2024-09-13T09:46:20.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-13T09:46:20.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:46:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:46:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:46:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:46:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:46:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 133 = '' 2024-09-13T09:46:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:46:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_8.status 2024-09-13T09:46:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:46:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_8.status 2024-09-13T09:46:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-13T09:46:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:46:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:46:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:46:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:46:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:46:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 133 = '' 2024-09-13T09:46:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:46:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_8.status 2024-09-13T09:46:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:46:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_8.status 2024-09-13T09:46:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-13T09:46:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=133 2024-09-13T09:46:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 133 = 133 2024-09-13T09:46:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-09-13T09:46:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:46:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:46:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:46:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:46:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-09-13T09:46:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_8': 2024-09-13T09:46:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:46:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:46:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:46:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 128233ed9444 2024-09-13T09:46:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.128233ed9444 2024-09-13T09:46:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:46:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:46:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:46:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:46:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:37:12 2024 2024-09-13T09:46:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:37:12 2024 2024-09-13T09:46:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:37:12 2024 2024-09-13T09:46:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:46:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:46:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:46:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 8a9c351e-2826-4d1c-9d73-5faec463921f 2024-09-13T09:46:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:46:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_8 snap 2024-09-13T09:46:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:46:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:46:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_8.export 2024-09-13T09:46:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_8.export 2024-09-13T09:46:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_8.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_8.export 2024-09-13T09:46:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_8.export 2024-09-13T09:46:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:46:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_8.export 2024-09-13T09:46:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:46:24.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_8.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_8.export 2024-09-13T09:46:24.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_8.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_8.export 2024-09-13T09:46:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:46:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_9 2024-09-13T09:46:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_9 snap 2024-09-13T09:46:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:46:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:46:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-09-13T09:46:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:46:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-09-13T09:46:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:46:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:46:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:46:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:46:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:46:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-09-13T09:46:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:46:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:46:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:46:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:46:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-09-13T09:46:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:46:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:46:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:46:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:46:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:46:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:46:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:46:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:46:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:46:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:46:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-09-13T09:46:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:46:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:46:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:46:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:46:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:46:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:46:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:46:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:46:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:46:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:46:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:46:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:46:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:46:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:46:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:46:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:46:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:46:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:46:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:46:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:46:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:46:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:46:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:46:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:46:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:46:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:46:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:46:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-09-13T09:46:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:46:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:46:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:46:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:46:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:46:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:46:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:46:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-09-13T09:46:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:46:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:46:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:46:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:46:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2024-09-13T09:46:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:46:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:46:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_9.status 2024-09-13T09:46:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_9.status 2024-09-13T09:46:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:46:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_9.status 2024-09-13T09:46:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_9.status 2024-09-13T09:46:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_9.status 2024-09-13T09:46:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:46:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_9.status 2024-09-13T09:46:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2024-09-13T09:46:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:46:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2024-09-13T09:46:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 135 2024-09-13T09:46:28.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_9.status 2024-09-13T09:46:28.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:28.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:46:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_9.status 2024-09-13T09:46:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2024-09-13T09:46:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:46:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=135 2024-09-13T09:46:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:46:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:46:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_9.status 2024-09-13T09:46:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:46:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_9.status 2024-09-13T09:46:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-13T09:46:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=76 2024-09-13T09:46:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 135 = 76 2024-09-13T09:46:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:46:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_9.status 2024-09-13T09:46:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:46:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_9.status 2024-09-13T09:46:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-13T09:46:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:46:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:46:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:46:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:46:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:46:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 135 = '' 2024-09-13T09:46:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:46:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_9.status 2024-09-13T09:46:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:46:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_9.status 2024-09-13T09:46:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-13T09:46:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=135 2024-09-13T09:46:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 135 = 135 2024-09-13T09:46:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-09-13T09:46:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:46:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:46:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:46:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:46:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-09-13T09:46:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_9': 2024-09-13T09:46:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:46:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:46:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:46:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 11f9ec72c4fb 2024-09-13T09:46:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.11f9ec72c4fb 2024-09-13T09:46:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:46:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:46:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:46:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:46:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:37:22 2024 2024-09-13T09:46:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:37:22 2024 2024-09-13T09:46:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:37:22 2024 2024-09-13T09:46:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:46:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:46:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:46:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: ce3d15a2-54a3-4901-a2c5-f75b935cc27e 2024-09-13T09:46:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:46:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_9 snap 2024-09-13T09:46:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:46:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:46:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_9.export 2024-09-13T09:46:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_9.export 2024-09-13T09:46:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_9.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_9.export 2024-09-13T09:46:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_9.export 2024-09-13T09:46:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:46:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_9.export 2024-09-13T09:46:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:46:31.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_9.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_9.export 2024-09-13T09:46:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_9.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_9.export 2024-09-13T09:46:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:46:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_10 2024-09-13T09:46:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_10 snap 2024-09-13T09:46:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:46:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:46:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-09-13T09:46:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:46:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-09-13T09:46:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:46:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:46:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:46:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:46:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:46:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-09-13T09:46:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:46:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:46:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:46:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:46:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-09-13T09:46:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:46:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:46:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:46:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:46:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:46:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:46:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:46:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:46:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:46:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:46:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-09-13T09:46:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:46:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:46:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:46:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:46:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:46:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:46:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:46:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:46:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:46:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:46:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:46:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:46:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:46:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:46:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:46:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:46:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:46:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:46:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:46:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:46:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:46:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:46:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:46:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:46:34.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:46:34.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:46:34.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:46:34.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-09-13T09:46:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-13T09:46:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:46:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:46:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:46:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2024-09-13T09:46:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:46:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:46:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:46:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:46:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:46:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:46:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:46:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:46:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:46:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:46:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:46:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:46:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:46:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:46:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:46:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:46:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:46:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:46:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:46:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:46:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:46:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:46:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:46:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:46:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:46:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:46:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:46:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:46:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:46:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:46:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2024-09-13T09:46:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-13T09:46:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:46:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:46:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:46:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2024-09-13T09:46:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:46:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:46:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:46:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:46:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:46:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:46:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:46:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:46:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:46:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:46:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:46:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:46:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:46:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:46:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:46:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:46:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:46:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:46:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:46:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:46:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:46:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:46:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:46:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:46:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:46:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:46:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:46:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:46:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:46:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:46:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2024-09-13T09:46:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-13T09:46:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:46:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:46:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:46:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_10 2024-09-13T09:46:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:46:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:46:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:46:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:46:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:46:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:46:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:46:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:46:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:46:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:46:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:46:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:46:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:46:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:46:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:46:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:46:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:46:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:46:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:46:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:46:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:46:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:46:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:46:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:46:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:46:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:46:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:46:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:46:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:46:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:46:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_10 2024-09-13T09:46:34.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:46:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:46:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:46:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:46:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:46:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:46:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:46:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-09-13T09:46:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:46:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:46:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:46:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:46:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2024-09-13T09:46:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:46:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:46:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_10.status 2024-09-13T09:46:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_10.status 2024-09-13T09:46:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:46:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_10.status 2024-09-13T09:46:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_10.status 2024-09-13T09:46:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_10.status 2024-09-13T09:46:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:46:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_10.status 2024-09-13T09:46:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2024-09-13T09:46:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:46:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2024-09-13T09:46:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 137 2024-09-13T09:46:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_10.status 2024-09-13T09:46:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:46:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_10.status 2024-09-13T09:46:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2024-09-13T09:46:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:35.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:46:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=137 2024-09-13T09:46:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:46:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:46:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_10.status 2024-09-13T09:46:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:46:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_10.status 2024-09-13T09:46:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-13T09:46:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=77 2024-09-13T09:46:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 137 = 77 2024-09-13T09:46:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:46:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_10.status 2024-09-13T09:46:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:46:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_10.status 2024-09-13T09:46:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-13T09:46:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:35.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:46:35.886 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:46:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:46:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:46:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:46:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 137 = '' 2024-09-13T09:46:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:46:36.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_10.status 2024-09-13T09:46:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:46:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_10.status 2024-09-13T09:46:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-13T09:46:36.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=137 2024-09-13T09:46:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 137 = 137 2024-09-13T09:46:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-09-13T09:46:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:46:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:46:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:46:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:46:37.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-09-13T09:46:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_10': 2024-09-13T09:46:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:46:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:46:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:46:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 114eb1a572c7 2024-09-13T09:46:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.114eb1a572c7 2024-09-13T09:46:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:46:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:46:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:46:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:46:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:37:30 2024 2024-09-13T09:46:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:37:30 2024 2024-09-13T09:46:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:37:30 2024 2024-09-13T09:46:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:46:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:46:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:46:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: dd6c9225-99be-46ca-b40c-23bfa9e8daa3 2024-09-13T09:46:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:46:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_10 snap 2024-09-13T09:46:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:46:37.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:37.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:46:37.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_10.export 2024-09-13T09:46:37.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_10.export 2024-09-13T09:46:37.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_10.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_10.export 2024-09-13T09:46:37.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_10.export 2024-09-13T09:46:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:46:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_10.export 2024-09-13T09:46:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:46:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_10.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_10.export 2024-09-13T09:46:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_10.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_10.export 2024-09-13T09:46:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:46:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_11 2024-09-13T09:46:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_11 snap 2024-09-13T09:46:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:46:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:46:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-09-13T09:46:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:46:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-09-13T09:46:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:46:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:46:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:46:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:46:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:46:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-09-13T09:46:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:46:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:46:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:46:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:46:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-09-13T09:46:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:46:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:46:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:46:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:46:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:46:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:46:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:46:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:46:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:46:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:46:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-09-13T09:46:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:46:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:46:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:46:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:46:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:46:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:46:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:46:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:46:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:46:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:46:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:46:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:46:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:46:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:46:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:46:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:46:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:46:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:46:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:46:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:46:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:46:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:46:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:46:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:46:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:46:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:46:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:46:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-09-13T09:46:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-09-13T09:46:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:46:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:46:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:46:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2024-09-13T09:46:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:46:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:46:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:46:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:46:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:46:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:46:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:46:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:46:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:46:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:46:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:46:41.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:46:41.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:46:41.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:46:41.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:46:41.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:46:41.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:46:41.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:46:41.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:46:41.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:46:41.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:41.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:46:41.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:46:41.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:46:41.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:46:41.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:46:41.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:46:41.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:46:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:46:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:46:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:46:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2024-09-13T09:46:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-09-13T09:46:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:46:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:46:41.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:46:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2024-09-13T09:46:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:46:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:46:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:46:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:46:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:46:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:46:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:46:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:46:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:46:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:46:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:46:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:46:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:46:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:46:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:46:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:46:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:46:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:46:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:46:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:46:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:46:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:46:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:46:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:46:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:46:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:46:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:46:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:46:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:46:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:46:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2024-09-13T09:46:41.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:46:41.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:46:41.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:46:41.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:46:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:46:41.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:46:41.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:46:41.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:41.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-09-13T09:46:41.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:46:41.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:41.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:41.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:46:41.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:46:41.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:46:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2024-09-13T09:46:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:46:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:46:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_11.status 2024-09-13T09:46:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_11.status 2024-09-13T09:46:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:46:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_11.status 2024-09-13T09:46:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_11.status 2024-09-13T09:46:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_11.status 2024-09-13T09:46:41.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:46:41.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_11.status 2024-09-13T09:46:41.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2024-09-13T09:46:41.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:41.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:41.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:46:41.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2024-09-13T09:46:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 139 2024-09-13T09:46:42.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_11.status 2024-09-13T09:46:42.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:42.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:42.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:46:42.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_11.status 2024-09-13T09:46:42.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2024-09-13T09:46:42.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:46:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=139 2024-09-13T09:46:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:46:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:46:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_11.status 2024-09-13T09:46:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:46:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_11.status 2024-09-13T09:46:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-13T09:46:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:42.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=78 2024-09-13T09:46:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 139 = 78 2024-09-13T09:46:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:46:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_11.status 2024-09-13T09:46:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:46:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_11.status 2024-09-13T09:46:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-13T09:46:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:46:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:46:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:46:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:46:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:46:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 139 = '' 2024-09-13T09:46:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:46:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_11.status 2024-09-13T09:46:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:46:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_11.status 2024-09-13T09:46:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-13T09:46:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=139 2024-09-13T09:46:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 139 = 139 2024-09-13T09:46:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-09-13T09:46:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:46:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:46:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:46:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:46:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-09-13T09:46:44.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_11': 2024-09-13T09:46:44.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:46:44.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:46:44.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:46:44.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 11f9bd5523af 2024-09-13T09:46:44.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.11f9bd5523af 2024-09-13T09:46:44.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:46:44.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:46:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:46:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:46:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:37:42 2024 2024-09-13T09:46:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:37:42 2024 2024-09-13T09:46:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:37:42 2024 2024-09-13T09:46:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:46:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:46:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:46:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: e78a9a0a-006b-4a90-8dde-2667b994e2f6 2024-09-13T09:46:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:46:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_11 snap 2024-09-13T09:46:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:46:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:46:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_11.export 2024-09-13T09:46:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_11.export 2024-09-13T09:46:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_11.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_11.export 2024-09-13T09:46:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_11.export 2024-09-13T09:46:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:46:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_11.export 2024-09-13T09:46:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:46:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_11.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_11.export 2024-09-13T09:46:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_11.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_11.export 2024-09-13T09:46:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:46:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_12 2024-09-13T09:46:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_12 snap 2024-09-13T09:46:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:46:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:46:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-09-13T09:46:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:46:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-09-13T09:46:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:46:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:46:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:46:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:46:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:46:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-09-13T09:46:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:46:48.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:46:48.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:46:48.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:46:48.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-09-13T09:46:48.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:46:48.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:46:48.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:48.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:46:48.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:46:48.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:46:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:46:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:46:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:46:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:46:48.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:46:48.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-09-13T09:46:48.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:46:48.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:48.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:46:48.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:46:48.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:46:48.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:46:48.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:46:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:46:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:46:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:46:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:46:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:46:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:46:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:46:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:46:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:46:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:46:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:46:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:46:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:46:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:46:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:46:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:46:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:46:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:46:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:46:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:46:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:46:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-09-13T09:46:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-13T09:46:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:46:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:46:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:46:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2024-09-13T09:46:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:46:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:46:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:46:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:46:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:46:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:46:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:46:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:46:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:46:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:46:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:46:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:46:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:46:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:46:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:46:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:46:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:46:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:46:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:46:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:46:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:46:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:46:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:46:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:46:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:46:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:46:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:46:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:46:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:46:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:46:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2024-09-13T09:46:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-13T09:46:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:46:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:46:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:46:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2024-09-13T09:46:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:46:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:46:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:46:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:46:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:46:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:46:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:46:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:46:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:46:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:46:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:46:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:46:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:46:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:46:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:46:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:46:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:46:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:46:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:46:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:46:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:46:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:46:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:46:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:46:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:46:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:46:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:46:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:46:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:46:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:46:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2024-09-13T09:46:48.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:46:48.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:46:48.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:46:48.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:46:48.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:46:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:46:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:46:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-09-13T09:46:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:46:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:46:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:46:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:46:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2024-09-13T09:46:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:46:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:46:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_12.status 2024-09-13T09:46:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_12.status 2024-09-13T09:46:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:46:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_12.status 2024-09-13T09:46:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_12.status 2024-09-13T09:46:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_12.status 2024-09-13T09:46:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:46:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_12.status 2024-09-13T09:46:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2024-09-13T09:46:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:46:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2024-09-13T09:46:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 141 2024-09-13T09:46:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_12.status 2024-09-13T09:46:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:46:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_12.status 2024-09-13T09:46:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2024-09-13T09:46:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:46:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=141 2024-09-13T09:46:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:46:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:46:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_12.status 2024-09-13T09:46:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:46:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_12.status 2024-09-13T09:46:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-13T09:46:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=79 2024-09-13T09:46:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 141 = 79 2024-09-13T09:46:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:46:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_12.status 2024-09-13T09:46:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:46:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_12.status 2024-09-13T09:46:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-13T09:46:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:46:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:46:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:46:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:46:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:46:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 141 = '' 2024-09-13T09:46:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:46:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_12.status 2024-09-13T09:46:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:46:50.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_12.status 2024-09-13T09:46:50.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-13T09:46:50.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=141 2024-09-13T09:46:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 141 = 141 2024-09-13T09:46:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-09-13T09:46:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:46:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:46:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:46:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:46:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-09-13T09:46:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_12': 2024-09-13T09:46:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:46:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:46:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:46:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12829c583b35 2024-09-13T09:46:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12829c583b35 2024-09-13T09:46:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:46:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:46:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:46:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:46:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:37:52 2024 2024-09-13T09:46:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:37:52 2024 2024-09-13T09:46:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:37:52 2024 2024-09-13T09:46:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:46:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:46:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:46:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: d677823f-dea5-4da8-94e6-42070cc16fc2 2024-09-13T09:46:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:46:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_12 snap 2024-09-13T09:46:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:46:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:46:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_12.export 2024-09-13T09:46:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_12.export 2024-09-13T09:46:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_12.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_12.export 2024-09-13T09:46:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_12.export 2024-09-13T09:46:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:46:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_12.export 2024-09-13T09:46:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:46:52.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_12.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_12.export 2024-09-13T09:46:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_12.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_12.export 2024-09-13T09:46:52.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:52.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:46:52.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_13 2024-09-13T09:46:52.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_13 snap 2024-09-13T09:46:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:46:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:46:52.533 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5881700475393691) > weight (0.25) 2024-09-13T09:46:52.534 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-13T09:46:52.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-13T09:46:52.535 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-13T09:46:52.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T09:46:52.534+0000 7f53f7e65640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 49256) UID: 1000 2024-09-13T09:46:53.535 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-13T09:46:53.535 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:46:53.536 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-13T09:46:53.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-13T09:46:53.536 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-13T09:46:53.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-13T09:46:53.540 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2024-09-13T09:46:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-09-13T09:46:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:46:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-09-13T09:46:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:46:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:46:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:46:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:46:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:46:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-09-13T09:46:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:46:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:46:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:46:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:46:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-09-13T09:46:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:46:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:46:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:46:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:46:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:46:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:46:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:46:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:46:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:46:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:46:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-09-13T09:46:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:46:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:46:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:46:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:46:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:46:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:46:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:46:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:46:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:46:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:46:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:46:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:46:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:46:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:46:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:46:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:46:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:46:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:46:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:46:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:46:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:46:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:46:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:46:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:46:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:46:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:46:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:46:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:46:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:46:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:46:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-09-13T09:46:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:46:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:46:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:46:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:46:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:46:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:46:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:46:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-09-13T09:46:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:46:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:46:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:46:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:46:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2024-09-13T09:46:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:46:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:55.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:46:55.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_13.status 2024-09-13T09:46:55.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_13.status 2024-09-13T09:46:55.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:46:55.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_13.status 2024-09-13T09:46:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_13.status 2024-09-13T09:46:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_13.status 2024-09-13T09:46:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:46:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_13.status 2024-09-13T09:46:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2024-09-13T09:46:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:46:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2024-09-13T09:46:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 143 2024-09-13T09:46:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_13.status 2024-09-13T09:46:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:46:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_13.status 2024-09-13T09:46:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2024-09-13T09:46:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:56.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:46:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=143 2024-09-13T09:46:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:46:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:46:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_13.status 2024-09-13T09:46:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:46:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_13.status 2024-09-13T09:46:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-13T09:46:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=80 2024-09-13T09:46:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 143 = 80 2024-09-13T09:46:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:46:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_13.status 2024-09-13T09:46:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:46:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_13.status 2024-09-13T09:46:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-13T09:46:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:46:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:46:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:46:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:46:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:46:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 143 = '' 2024-09-13T09:46:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:46:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:46:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_13.status 2024-09-13T09:46:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:46:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_13.status 2024-09-13T09:46:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-13T09:46:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:46:57.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:46:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=143 2024-09-13T09:46:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 143 = 143 2024-09-13T09:46:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-09-13T09:46:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:46:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:46:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:46:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:46:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:46:58.633 INFO:tasks.ceph.cluster1.osd.0.smithi122.stderr:2024-09-13T09:46:58.632+0000 7ff198c84640 -1 reset not still connected to 0x560ff4638d00 2024-09-13T09:46:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-09-13T09:46:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_13': 2024-09-13T09:46:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:46:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:46:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:46:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c5fa59bbda 2024-09-13T09:46:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c5fa59bbda 2024-09-13T09:46:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:46:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:46:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:46:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:46:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:38:06 2024 2024-09-13T09:46:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:38:06 2024 2024-09-13T09:46:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:38:06 2024 2024-09-13T09:46:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:46:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:46:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:46:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: fa86eec0-62a5-41fc-9cc3-aa9334a25549 2024-09-13T09:46:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:46:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_13 snap 2024-09-13T09:46:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:46:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:46:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_13.export 2024-09-13T09:46:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_13.export 2024-09-13T09:46:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_13.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_13.export 2024-09-13T09:46:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_13.export 2024-09-13T09:46:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:46:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_13.export 2024-09-13T09:46:59.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:46:59.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_13.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_13.export 2024-09-13T09:46:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_13.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_13.export 2024-09-13T09:46:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:46:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:46:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_14 2024-09-13T09:46:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_14 snap 2024-09-13T09:46:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:46:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:46:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:46:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:46:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:47:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-09-13T09:47:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:47:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-09-13T09:47:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:47:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:47:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:47:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:47:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:47:02.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-09-13T09:47:02.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:02.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:02.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:47:02.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:47:02.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:47:02.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:47:02.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-09-13T09:47:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:47:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:47:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:47:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:47:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:47:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:47:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:47:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:47:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:47:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-09-13T09:47:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:47:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:47:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:47:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:47:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:47:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:47:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:47:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:47:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:47:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:47:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:47:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:47:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:47:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:47:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:47:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:47:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-09-13T09:47:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:47:02.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:47:02.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:47:02.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:47:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:47:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:47:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:47:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-09-13T09:47:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:47:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:47:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:47:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:47:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2024-09-13T09:47:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:47:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:47:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_14.status 2024-09-13T09:47:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_14.status 2024-09-13T09:47:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:47:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_14.status 2024-09-13T09:47:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_14.status 2024-09-13T09:47:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_14.status 2024-09-13T09:47:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:47:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_14.status 2024-09-13T09:47:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2024-09-13T09:47:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:47:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2024-09-13T09:47:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 145 2024-09-13T09:47:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_14.status 2024-09-13T09:47:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:47:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_14.status 2024-09-13T09:47:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2024-09-13T09:47:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:47:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=145 2024-09-13T09:47:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:47:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:47:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_14.status 2024-09-13T09:47:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:47:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_14.status 2024-09-13T09:47:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-13T09:47:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=81 2024-09-13T09:47:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 145 = 81 2024-09-13T09:47:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:47:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_14.status 2024-09-13T09:47:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:47:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_14.status 2024-09-13T09:47:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-13T09:47:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:47:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:47:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:47:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:47:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:47:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 145 = '' 2024-09-13T09:47:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:47:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_14.status 2024-09-13T09:47:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:47:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_14.status 2024-09-13T09:47:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-13T09:47:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=145 2024-09-13T09:47:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 145 = 145 2024-09-13T09:47:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-09-13T09:47:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:47:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:47:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:47:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:47:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:47:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-09-13T09:47:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_14': 2024-09-13T09:47:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:47:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:47:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:47:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 114e729693d1 2024-09-13T09:47:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.114e729693d1 2024-09-13T09:47:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:47:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:47:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:47:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:47:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:38:13 2024 2024-09-13T09:47:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:38:13 2024 2024-09-13T09:47:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:38:13 2024 2024-09-13T09:47:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:47:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:47:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:47:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 393ab671-e876-42d3-878c-4b6c73393fe3 2024-09-13T09:47:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:47:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_14 snap 2024-09-13T09:47:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:47:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:47:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:47:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_14.export 2024-09-13T09:47:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_14.export 2024-09-13T09:47:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_14.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_14.export 2024-09-13T09:47:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_14.export 2024-09-13T09:47:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:47:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_14.export 2024-09-13T09:47:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:47:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_14.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_14.export 2024-09-13T09:47:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_14.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_14.export 2024-09-13T09:47:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:47:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_15 2024-09-13T09:47:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_15 snap 2024-09-13T09:47:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:06.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:06.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:47:06.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:47:06.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:47:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-09-13T09:47:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:47:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-09-13T09:47:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:47:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:47:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:47:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:47:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:47:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-09-13T09:47:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:47:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:47:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:47:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:47:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-09-13T09:47:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:47:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:47:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:47:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:47:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:47:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:47:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:47:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:47:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:47:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-09-13T09:47:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:47:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:47:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:47:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:47:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:47:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:47:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:47:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:47:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:47:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:47:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:47:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:47:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:47:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:47:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:47:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:47:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-09-13T09:47:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:47:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:47:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:47:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:47:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:47:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:47:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:47:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-09-13T09:47:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:47:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:47:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:47:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:47:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2024-09-13T09:47:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:47:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:47:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_15.status 2024-09-13T09:47:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_15.status 2024-09-13T09:47:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:47:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_15.status 2024-09-13T09:47:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_15.status 2024-09-13T09:47:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_15.status 2024-09-13T09:47:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:47:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_15.status 2024-09-13T09:47:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2024-09-13T09:47:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:47:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2024-09-13T09:47:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 147 2024-09-13T09:47:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_15.status 2024-09-13T09:47:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:47:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_15.status 2024-09-13T09:47:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2024-09-13T09:47:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:10.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:47:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=147 2024-09-13T09:47:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:47:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_15.status 2024-09-13T09:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_15.status 2024-09-13T09:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-13T09:47:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=82 2024-09-13T09:47:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 147 = 82 2024-09-13T09:47:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:47:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_15.status 2024-09-13T09:47:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:47:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_15.status 2024-09-13T09:47:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-13T09:47:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:47:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:47:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:47:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:47:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:47:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 147 = '' 2024-09-13T09:47:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:47:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_15.status 2024-09-13T09:47:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:47:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_15.status 2024-09-13T09:47:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-13T09:47:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=147 2024-09-13T09:47:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 147 = 147 2024-09-13T09:47:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-09-13T09:47:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:47:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:47:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:47:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:47:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:47:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-09-13T09:47:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_15': 2024-09-13T09:47:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:47:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:47:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:47:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c52bf367f7 2024-09-13T09:47:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c52bf367f7 2024-09-13T09:47:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:47:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:47:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:47:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:47:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:38:48 2024 2024-09-13T09:47:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:38:48 2024 2024-09-13T09:47:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:38:48 2024 2024-09-13T09:47:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:47:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:47:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:47:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 22b97631-339a-40e4-93c4-03fc33081710 2024-09-13T09:47:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:47:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_15 snap 2024-09-13T09:47:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:47:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:47:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:47:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_15.export 2024-09-13T09:47:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_15.export 2024-09-13T09:47:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_15.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_15.export 2024-09-13T09:47:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_15.export 2024-09-13T09:47:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:47:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_15.export 2024-09-13T09:47:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:47:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_15.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_15.export 2024-09-13T09:47:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_15.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_15.export 2024-09-13T09:47:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:47:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_16 2024-09-13T09:47:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_16 snap 2024-09-13T09:47:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:47:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:47:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:47:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-09-13T09:47:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:47:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-09-13T09:47:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:47:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:47:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:47:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:47:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:47:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-09-13T09:47:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:47:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:47:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:47:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:47:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-09-13T09:47:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:47:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:47:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:47:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:47:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:47:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:47:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:47:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:47:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:47:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-09-13T09:47:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:47:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:47:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:47:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:47:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:47:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:47:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:47:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:47:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:47:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:47:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:47:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:47:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:47:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:47:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:47:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:47:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-09-13T09:47:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-13T09:47:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:47:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:47:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-09-13T09:47:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:47:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:47:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:47:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:47:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:47:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:47:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:47:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:47:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:47:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:47:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:47:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:47:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:47:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:47:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:47:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:47:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:47:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:47:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:47:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-09-13T09:47:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-13T09:47:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:47:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:47:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2024-09-13T09:47:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:47:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:47:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:47:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:47:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:47:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:47:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:47:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:47:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:47:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:47:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:47:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:47:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:47:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:47:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:47:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:47:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:47:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:47:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:47:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2024-09-13T09:47:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-13T09:47:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:47:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:47:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_16 2024-09-13T09:47:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:47:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:47:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:47:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:47:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:47:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:47:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:47:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:47:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:47:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:47:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:47:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:47:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:47:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:47:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:47:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:47:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:47:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:47:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:47:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_16 2024-09-13T09:47:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-13T09:47:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:47:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:47:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:47:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 1 2024-09-13T09:47:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:47:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:47:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-09-13T09:47:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:47:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:47:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:47:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:47:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:47:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:47:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:47:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:47:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:47:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:47:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:47:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:47:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:47:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:47:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:47:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:47:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-09-13T09:47:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-13T09:47:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:47:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:47:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-09-13T09:47:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:47:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:47:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:47:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:47:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:47:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:47:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:47:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:47:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:47:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:47:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:47:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:47:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:47:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:47:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:47:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:47:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:47:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:47:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:47:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-09-13T09:47:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-13T09:47:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:47:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:47:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2024-09-13T09:47:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:47:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:47:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:47:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:47:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:47:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:47:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:47:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:47:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:47:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:47:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:47:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:47:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:47:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:47:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:47:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:47:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:47:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:47:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:47:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2024-09-13T09:47:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-13T09:47:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:47:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:47:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_16 2024-09-13T09:47:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:47:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:47:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:47:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:47:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:47:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:47:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:47:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:47:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:47:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:47:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:47:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:17.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:17.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:47:17.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:47:17.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:17.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:17.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:47:17.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:47:17.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:47:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:47:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:47:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:47:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_16 2024-09-13T09:47:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-13T09:47:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:47:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:47:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:47:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 2 2024-09-13T09:47:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:47:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:47:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-09-13T09:47:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:47:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:47:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:47:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:47:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:47:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:47:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:47:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:47:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:47:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:47:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:47:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:47:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:47:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:47:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:47:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:47:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-09-13T09:47:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-13T09:47:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:47:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:47:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-09-13T09:47:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:47:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:47:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:47:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:47:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:47:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:47:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:47:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:47:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:47:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:47:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:47:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:47:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:47:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:47:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:47:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:47:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:47:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:47:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:47:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-09-13T09:47:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-13T09:47:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:47:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:47:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2024-09-13T09:47:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:47:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:47:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:47:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:47:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:47:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:47:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:47:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:47:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:47:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:47:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:47:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:19.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:47:19.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:47:19.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:19.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:19.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:47:19.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:47:19.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:47:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:47:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:47:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:47:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2024-09-13T09:47:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-13T09:47:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:47:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:47:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_16 2024-09-13T09:47:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:47:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:47:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:47:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:47:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:47:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:47:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:47:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:47:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:47:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:47:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:47:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:47:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:47:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:47:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:47:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:47:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:47:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:47:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:47:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_16 2024-09-13T09:47:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-13T09:47:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:47:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:47:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:47:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 4 2024-09-13T09:47:23.431 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9430187243055469) > weight (0.25) 2024-09-13T09:47:23.432 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-13T09:47:23.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-13T09:47:23.432 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-13T09:47:23.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:47:23.432+0000 7fc85555f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 49612) UID: 1000 2024-09-13T09:47:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:47:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:47:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-09-13T09:47:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:47:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:47:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:47:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:47:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:47:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:47:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:47:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:47:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:47:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:47:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:47:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:47:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:47:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:47:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:47:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:47:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-09-13T09:47:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-13T09:47:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:47:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:47:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-09-13T09:47:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:47:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:47:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:47:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:47:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:47:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:47:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:47:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:47:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:47:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:47:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:47:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:47:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:47:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:47:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:47:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:47:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:47:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:47:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:47:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-09-13T09:47:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-13T09:47:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:47:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:47:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2024-09-13T09:47:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:47:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:47:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:47:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:47:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:47:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:47:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:47:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:47:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:47:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:47:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:47:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:47:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:47:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:47:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:47:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:47:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:47:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:47:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:47:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2024-09-13T09:47:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-13T09:47:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:47:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:47:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_16 2024-09-13T09:47:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:47:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:47:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:47:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:47:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:47:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:47:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:47:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:47:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:47:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:47:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:47:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:47:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:47:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:47:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:47:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:47:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:47:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:47:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:47:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_16 2024-09-13T09:47:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:47:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:47:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:47:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:47:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:47:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:47:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:47:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-09-13T09:47:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:47:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:47:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:47:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:47:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2024-09-13T09:47:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:47:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:47:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_16.status 2024-09-13T09:47:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_16.status 2024-09-13T09:47:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:47:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_16.status 2024-09-13T09:47:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_16.status 2024-09-13T09:47:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_16.status 2024-09-13T09:47:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:47:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_16.status 2024-09-13T09:47:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2024-09-13T09:47:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:47:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2024-09-13T09:47:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 149 2024-09-13T09:47:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_16.status 2024-09-13T09:47:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:47:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_16.status 2024-09-13T09:47:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2024-09-13T09:47:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:47:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=149 2024-09-13T09:47:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:47:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:47:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_16.status 2024-09-13T09:47:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:47:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_16.status 2024-09-13T09:47:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-13T09:47:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:47:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:47:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:47:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:47:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:47:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 149 = '' 2024-09-13T09:47:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:47:25.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_16.status 2024-09-13T09:47:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:47:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_16.status 2024-09-13T09:47:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-13T09:47:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:47:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:47:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:47:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:47:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:47:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 149 = '' 2024-09-13T09:47:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:47:26.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_16.status 2024-09-13T09:47:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:47:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_16.status 2024-09-13T09:47:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-13T09:47:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:26.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=149 2024-09-13T09:47:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 149 = 149 2024-09-13T09:47:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-09-13T09:47:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:47:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:47:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:47:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:47:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:47:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-09-13T09:47:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_16': 2024-09-13T09:47:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:47:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:47:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:47:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 128b827799d1 2024-09-13T09:47:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.128b827799d1 2024-09-13T09:47:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:47:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:47:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:47:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:47:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:38:32 2024 2024-09-13T09:47:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:38:32 2024 2024-09-13T09:47:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:38:32 2024 2024-09-13T09:47:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:47:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:47:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:47:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: cd59b635-c822-4dda-a6e6-0e788a2c4725 2024-09-13T09:47:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:47:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_16 snap 2024-09-13T09:47:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:47:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:47:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:47:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_16.export 2024-09-13T09:47:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_16.export 2024-09-13T09:47:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_16.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_16.export 2024-09-13T09:47:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_16.export 2024-09-13T09:47:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:47:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_16.export 2024-09-13T09:47:27.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:47:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_16.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_16.export 2024-09-13T09:47:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_16.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_16.export 2024-09-13T09:47:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:47:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_17 2024-09-13T09:47:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_17 snap 2024-09-13T09:47:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:47:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:47:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:47:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-09-13T09:47:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:47:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-09-13T09:47:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:47:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:47:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:47:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:47:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:47:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-09-13T09:47:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:47:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:47:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:47:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:47:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-09-13T09:47:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:47:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:47:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:47:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:47:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:47:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:47:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:47:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:47:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:47:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-09-13T09:47:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:47:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:47:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:47:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:47:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:47:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:47:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:47:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:47:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:47:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:47:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:47:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:47:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:47:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:47:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:47:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:47:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-09-13T09:47:30.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-09-13T09:47:30.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:47:30.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:47:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2024-09-13T09:47:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:47:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:47:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:47:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:47:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:47:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:47:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:47:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:47:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:47:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:47:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:47:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:47:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:47:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:47:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:47:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:47:30.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:47:30.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:47:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:47:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2024-09-13T09:47:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-09-13T09:47:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:47:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:47:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_17 2024-09-13T09:47:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:47:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:47:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:47:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:47:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:47:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:47:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:47:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:47:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:47:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:47:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:47:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:47:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:47:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:47:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:47:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:47:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:47:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:47:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:47:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_17 2024-09-13T09:47:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:47:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:47:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:47:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:47:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:47:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:47:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:47:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-09-13T09:47:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:47:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:47:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:47:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:47:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2024-09-13T09:47:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:47:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:47:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_17.status 2024-09-13T09:47:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_17.status 2024-09-13T09:47:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:47:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_17.status 2024-09-13T09:47:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_17.status 2024-09-13T09:47:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_17.status 2024-09-13T09:47:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:47:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_17.status 2024-09-13T09:47:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2024-09-13T09:47:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:47:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2024-09-13T09:47:31.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 151 2024-09-13T09:47:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_17.status 2024-09-13T09:47:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:47:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_17.status 2024-09-13T09:47:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2024-09-13T09:47:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:47:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=151 2024-09-13T09:47:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:47:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:47:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_17.status 2024-09-13T09:47:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:47:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_17.status 2024-09-13T09:47:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-13T09:47:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:47:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:47:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:47:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:47:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:47:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 151 = '' 2024-09-13T09:47:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:47:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_17.status 2024-09-13T09:47:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:47:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_17.status 2024-09-13T09:47:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-13T09:47:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:32.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:47:32.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:32.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:47:32.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:47:32.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:47:32.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:47:32.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 151 = '' 2024-09-13T09:47:32.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:32.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:47:32.433 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-13T09:47:32.434 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:47:32.434 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-13T09:47:32.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-13T09:47:32.434 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-13T09:47:32.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-13T09:47:32.437 INFO:tasks.rbd_mirror_thrash:waiting for 112 secs before thrashing 2024-09-13T09:47:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_17.status 2024-09-13T09:47:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:47:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_17.status 2024-09-13T09:47:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-13T09:47:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=151 2024-09-13T09:47:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 151 = 151 2024-09-13T09:47:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-09-13T09:47:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:47:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:47:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:47:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:47:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:47:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-09-13T09:47:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_17': 2024-09-13T09:47:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:47:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:47:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:47:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 128b3828d82d 2024-09-13T09:47:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.128b3828d82d 2024-09-13T09:47:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:47:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:47:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:47:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:47:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:38:42 2024 2024-09-13T09:47:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:38:42 2024 2024-09-13T09:47:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:38:42 2024 2024-09-13T09:47:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:47:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:47:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:47:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: fc6f16a3-057d-4588-8097-7af12a77aa44 2024-09-13T09:47:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:47:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_17 snap 2024-09-13T09:47:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:47:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:47:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:47:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_17.export 2024-09-13T09:47:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_17.export 2024-09-13T09:47:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_17.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_17.export 2024-09-13T09:47:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_17.export 2024-09-13T09:47:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:47:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_17.export 2024-09-13T09:47:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:47:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_17.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_17.export 2024-09-13T09:47:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_17.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_17.export 2024-09-13T09:47:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:47:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_18 2024-09-13T09:47:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_18 snap 2024-09-13T09:47:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:47:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:47:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:47:36.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-09-13T09:47:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:47:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-09-13T09:47:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:47:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:47:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:47:36.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:47:36.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:47:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-09-13T09:47:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:47:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:47:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:47:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:47:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-09-13T09:47:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:47:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:47:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:47:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:47:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:47:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:47:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:47:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:47:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:47:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-09-13T09:47:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:47:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:47:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:47:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:47:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:47:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:47:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:47:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:47:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:47:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:37.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:47:37.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:47:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:47:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:47:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:47:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:47:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:47:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-09-13T09:47:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-13T09:47:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:47:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:47:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2024-09-13T09:47:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:47:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:47:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:47:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:47:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:47:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:47:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:47:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:47:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:47:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:47:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:47:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:47:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:47:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:47:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:47:37.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:47:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:47:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:47:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:47:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2024-09-13T09:47:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-13T09:47:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:47:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:47:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2024-09-13T09:47:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:47:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:47:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:47:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:47:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:47:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:47:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:47:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:47:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:47:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:47:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:47:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:47:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:47:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:47:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:47:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:47:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:47:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:47:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:47:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2024-09-13T09:47:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:47:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:47:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:47:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:47:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:47:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:47:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:47:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-09-13T09:47:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:47:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:47:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:47:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:47:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2024-09-13T09:47:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:47:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:47:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_18.status 2024-09-13T09:47:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_18.status 2024-09-13T09:47:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:47:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_18.status 2024-09-13T09:47:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_18.status 2024-09-13T09:47:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_18.status 2024-09-13T09:47:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:47:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_18.status 2024-09-13T09:47:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2024-09-13T09:47:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:47:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2024-09-13T09:47:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 153 2024-09-13T09:47:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_18.status 2024-09-13T09:47:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:47:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_18.status 2024-09-13T09:47:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2024-09-13T09:47:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:47:38.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=153 2024-09-13T09:47:38.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:47:38.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:38.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:47:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_18.status 2024-09-13T09:47:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:47:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_18.status 2024-09-13T09:47:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-13T09:47:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=85 2024-09-13T09:47:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 153 = 85 2024-09-13T09:47:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:47:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_18.status 2024-09-13T09:47:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:47:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_18.status 2024-09-13T09:47:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-13T09:47:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:47:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:47:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:47:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:47:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:47:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 153 = '' 2024-09-13T09:47:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:47:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_18.status 2024-09-13T09:47:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:47:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_18.status 2024-09-13T09:47:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:39.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-13T09:47:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:47:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:47:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:47:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:47:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:47:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 153 = '' 2024-09-13T09:47:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:47:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_18.status 2024-09-13T09:47:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:47:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_18.status 2024-09-13T09:47:41.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-13T09:47:41.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=153 2024-09-13T09:47:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 153 = 153 2024-09-13T09:47:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-09-13T09:47:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:47:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:47:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:47:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:47:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:47:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-09-13T09:47:42.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_18': 2024-09-13T09:47:42.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:47:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:47:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:47:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 114ea9ad2f37 2024-09-13T09:47:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.114ea9ad2f37 2024-09-13T09:47:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:47:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:47:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:47:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:47:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:38:49 2024 2024-09-13T09:47:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:38:49 2024 2024-09-13T09:47:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:38:49 2024 2024-09-13T09:47:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:47:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:47:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:47:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: d7e04b0a-b54c-4532-b814-c9ae04053bfa 2024-09-13T09:47:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:47:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_18 snap 2024-09-13T09:47:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:47:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:47:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:47:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_18.export 2024-09-13T09:47:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_18.export 2024-09-13T09:47:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_18.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_18.export 2024-09-13T09:47:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_18.export 2024-09-13T09:47:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:47:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_18.export 2024-09-13T09:47:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:47:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_18.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_18.export 2024-09-13T09:47:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_18.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_18.export 2024-09-13T09:47:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:47:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_19 2024-09-13T09:47:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_19 snap 2024-09-13T09:47:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:47:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:47:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:47:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-09-13T09:47:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:47:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-09-13T09:47:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:47:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:47:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:47:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:47:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:47:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-09-13T09:47:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:47:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:47:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:47:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:47:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-09-13T09:47:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:47:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:47:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:47:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:47:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:47:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:47:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:47:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:47:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:47:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-09-13T09:47:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:47:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:47:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:47:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:47:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:47:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:47:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:47:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:47:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:47:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:47:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:47:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:47:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:47:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:47:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:47:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:47:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-09-13T09:47:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:47:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:47:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:47:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:47:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:47:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:47:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:47:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-09-13T09:47:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:47:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:47:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:47:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:47:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2024-09-13T09:47:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:47:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:47:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_19.status 2024-09-13T09:47:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_19.status 2024-09-13T09:47:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:47:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_19.status 2024-09-13T09:47:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_19.status 2024-09-13T09:47:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_19.status 2024-09-13T09:47:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:47:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_19.status 2024-09-13T09:47:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2024-09-13T09:47:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:47:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2024-09-13T09:47:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 155 2024-09-13T09:47:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_19.status 2024-09-13T09:47:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:47:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_19.status 2024-09-13T09:47:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2024-09-13T09:47:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:47:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=155 2024-09-13T09:47:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:47:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:47:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_19.status 2024-09-13T09:47:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:47:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_19.status 2024-09-13T09:47:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-13T09:47:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:46.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=86 2024-09-13T09:47:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 155 = 86 2024-09-13T09:47:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:47:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_19.status 2024-09-13T09:47:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:47:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_19.status 2024-09-13T09:47:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-13T09:47:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:46.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:47:46.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:47:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:47:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:47:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:47:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 155 = '' 2024-09-13T09:47:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:47:47.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_19.status 2024-09-13T09:47:47.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:47.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:47:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_19.status 2024-09-13T09:47:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-13T09:47:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=155 2024-09-13T09:47:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 155 = 155 2024-09-13T09:47:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-09-13T09:47:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:47:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:47:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:47:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:47:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:47:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-09-13T09:47:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_19': 2024-09-13T09:47:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:47:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:47:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:47:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12827fee299d 2024-09-13T09:47:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12827fee299d 2024-09-13T09:47:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:47:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:47:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:47:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:47:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:39:05 2024 2024-09-13T09:47:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:39:05 2024 2024-09-13T09:47:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:39:05 2024 2024-09-13T09:47:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:47:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:47:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:47:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 98a2a84e-2461-434e-b3e3-a5de2949eb05 2024-09-13T09:47:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:47:49.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:49.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_19 snap 2024-09-13T09:47:49.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:49.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:47:49.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:47:49.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:47:49.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_19.export 2024-09-13T09:47:49.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_19.export 2024-09-13T09:47:49.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_19.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_19.export 2024-09-13T09:47:49.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_19.export 2024-09-13T09:47:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:47:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_19.export 2024-09-13T09:47:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:47:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_19.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_19.export 2024-09-13T09:47:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_19.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_19.export 2024-09-13T09:47:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:47:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_20 2024-09-13T09:47:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_20 snap 2024-09-13T09:47:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:47:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:47:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:47:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-09-13T09:47:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:47:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-09-13T09:47:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:47:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:47:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:47:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:47:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:47:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-09-13T09:47:52.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:52.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:52.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:47:52.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:47:52.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:47:52.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:47:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-09-13T09:47:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:47:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:47:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:47:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:47:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:47:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:47:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:47:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:47:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:47:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-09-13T09:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:47:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:47:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:47:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:47:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:47:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:47:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:47:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:47:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:47:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:47:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:47:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:47:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-09-13T09:47:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-13T09:47:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:47:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:47:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-09-13T09:47:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:47:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:47:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:47:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:47:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:47:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:47:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:47:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:47:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:47:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:47:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:47:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:47:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:47:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:47:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:47:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:47:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:47:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:47:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:47:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-09-13T09:47:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-13T09:47:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:47:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:47:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:47:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2024-09-13T09:47:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:47:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:47:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:47:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:47:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:47:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:47:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:47:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:47:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:47:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:47:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:47:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:47:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:47:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:47:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:47:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:47:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:47:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:47:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:47:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:47:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:47:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:47:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:47:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:47:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:47:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:47:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:47:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:47:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:47:52.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:47:52.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:47:52.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:47:52.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2024-09-13T09:47:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:47:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:47:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:47:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:47:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:47:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:47:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:47:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-09-13T09:47:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:47:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:47:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:47:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:47:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2024-09-13T09:47:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:47:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:47:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_20.status 2024-09-13T09:47:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_20.status 2024-09-13T09:47:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:47:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_20.status 2024-09-13T09:47:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_20.status 2024-09-13T09:47:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_20.status 2024-09-13T09:47:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:47:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_20.status 2024-09-13T09:47:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2024-09-13T09:47:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:47:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2024-09-13T09:47:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 157 2024-09-13T09:47:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_20.status 2024-09-13T09:47:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:47:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_20.status 2024-09-13T09:47:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2024-09-13T09:47:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:47:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=157 2024-09-13T09:47:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:47:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:47:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_20.status 2024-09-13T09:47:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:47:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_20.status 2024-09-13T09:47:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-13T09:47:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=87 2024-09-13T09:47:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 157 = 87 2024-09-13T09:47:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:47:53.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_20.status 2024-09-13T09:47:53.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:53.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:47:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_20.status 2024-09-13T09:47:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-13T09:47:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:47:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:47:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:47:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:47:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:47:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 157 = '' 2024-09-13T09:47:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:47:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_20.status 2024-09-13T09:47:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:47:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_20.status 2024-09-13T09:47:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-13T09:47:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:47:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:47:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:47:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:47:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:47:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 157 = '' 2024-09-13T09:47:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:47:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:47:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_20.status 2024-09-13T09:47:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:47:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_20.status 2024-09-13T09:47:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-13T09:47:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:47:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:47:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=157 2024-09-13T09:47:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 157 = 157 2024-09-13T09:47:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-09-13T09:47:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:47:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:47:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:47:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:47:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:47:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-09-13T09:47:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_20': 2024-09-13T09:47:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:47:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:47:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:47:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 128bb4af823d 2024-09-13T09:47:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.128bb4af823d 2024-09-13T09:47:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:47:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:47:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:47:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:47:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:39:18 2024 2024-09-13T09:47:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:39:18 2024 2024-09-13T09:47:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:39:18 2024 2024-09-13T09:47:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:47:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:47:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:47:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: ab0efa63-cc31-4345-9caf-c5a58d606330 2024-09-13T09:47:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:47:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_20 snap 2024-09-13T09:47:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:47:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:47:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:47:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_20.export 2024-09-13T09:47:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_20.export 2024-09-13T09:47:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_20.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_20.export 2024-09-13T09:47:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_20.export 2024-09-13T09:47:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:47:57.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_20.export 2024-09-13T09:47:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:47:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_20.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_20.export 2024-09-13T09:47:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_20.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_20.export 2024-09-13T09:47:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:47:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:47:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_21 2024-09-13T09:47:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_21 snap 2024-09-13T09:47:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:47:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:47:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:47:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:47:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-09-13T09:47:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:47:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-09-13T09:47:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:47:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:47:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:47:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:47:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:47:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:47:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:48:00.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-09-13T09:48:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:48:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:48:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:48:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:48:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-09-13T09:48:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:48:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:48:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:48:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:48:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:48:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:48:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:48:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:48:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:48:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-09-13T09:48:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:48:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:48:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:48:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:48:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:48:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:48:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:48:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:48:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:48:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:48:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:48:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:48:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:48:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:48:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:48:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:48:00.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:00.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:00.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:00.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-09-13T09:48:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:48:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:48:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:48:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:48:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:48:00.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:48:00.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:48:00.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:00.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-09-13T09:48:00.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:48:00.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:00.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:00.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:48:00.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:48:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:48:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2024-09-13T09:48:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:48:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:48:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_21.status 2024-09-13T09:48:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_21.status 2024-09-13T09:48:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:48:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_21.status 2024-09-13T09:48:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_21.status 2024-09-13T09:48:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_21.status 2024-09-13T09:48:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:48:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_21.status 2024-09-13T09:48:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2024-09-13T09:48:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:48:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2024-09-13T09:48:01.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 159 2024-09-13T09:48:01.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_21.status 2024-09-13T09:48:01.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:01.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:01.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:48:01.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_21.status 2024-09-13T09:48:01.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2024-09-13T09:48:01.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:48:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=159 2024-09-13T09:48:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:48:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:48:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_21.status 2024-09-13T09:48:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:48:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_21.status 2024-09-13T09:48:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-13T09:48:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=88 2024-09-13T09:48:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 159 = 88 2024-09-13T09:48:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:48:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_21.status 2024-09-13T09:48:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:48:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_21.status 2024-09-13T09:48:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-13T09:48:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:48:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:48:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:48:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:48:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:48:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 159 = '' 2024-09-13T09:48:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:48:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_21.status 2024-09-13T09:48:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:48:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_21.status 2024-09-13T09:48:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-13T09:48:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:48:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:48:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:48:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:48:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:48:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 159 = '' 2024-09-13T09:48:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:48:03.541 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6244940051581485) > weight (0.25) 2024-09-13T09:48:03.541 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-13T09:48:03.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-13T09:48:03.542 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-13T09:48:03.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T09:48:03.542+0000 7fa10c885640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 53162) UID: 1000 2024-09-13T09:48:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_21.status 2024-09-13T09:48:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:48:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_21.status 2024-09-13T09:48:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-13T09:48:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:04.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=159 2024-09-13T09:48:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 159 = 159 2024-09-13T09:48:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-09-13T09:48:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:48:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:48:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:48:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:48:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:48:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-09-13T09:48:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_21': 2024-09-13T09:48:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:48:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:48:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:48:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c51c7659ed 2024-09-13T09:48:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c51c7659ed 2024-09-13T09:48:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:48:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:48:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:48:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:48:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:39:28 2024 2024-09-13T09:48:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:39:28 2024 2024-09-13T09:48:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:39:28 2024 2024-09-13T09:48:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:48:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:48:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:48:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 47058edf-e75b-483b-aaa7-812779184a08 2024-09-13T09:48:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:48:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_21 snap 2024-09-13T09:48:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:48:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:48:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:48:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_21.export 2024-09-13T09:48:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_21.export 2024-09-13T09:48:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_21.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_21.export 2024-09-13T09:48:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_21.export 2024-09-13T09:48:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:48:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_21.export 2024-09-13T09:48:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:48:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_21.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_21.export 2024-09-13T09:48:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_21.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_21.export 2024-09-13T09:48:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:48:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_22 2024-09-13T09:48:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_22 snap 2024-09-13T09:48:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:48:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:48:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:48:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-09-13T09:48:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:48:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-09-13T09:48:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:48:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:48:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:48:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:48:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:48:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-09-13T09:48:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:48:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:48:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:48:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:48:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-09-13T09:48:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:48:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:48:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:48:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:48:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:48:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:48:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:48:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:48:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:48:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-09-13T09:48:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:48:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:48:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:48:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:48:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:48:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:48:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:48:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:48:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:48:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:48:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:48:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:48:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:48:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:48:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:48:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:48:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-09-13T09:48:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:48:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:48:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:48:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:48:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:48:08.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:48:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:48:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-09-13T09:48:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:48:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:48:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:48:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:48:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2024-09-13T09:48:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:48:08.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:08.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:08.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:48:08.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_22.status 2024-09-13T09:48:08.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_22.status 2024-09-13T09:48:08.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:48:08.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_22.status 2024-09-13T09:48:08.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_22.status 2024-09-13T09:48:08.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_22.status 2024-09-13T09:48:08.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:48:08.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_22.status 2024-09-13T09:48:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2024-09-13T09:48:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:48:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2024-09-13T09:48:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 161 2024-09-13T09:48:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_22.status 2024-09-13T09:48:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:48:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_22.status 2024-09-13T09:48:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2024-09-13T09:48:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:48:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=161 2024-09-13T09:48:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:48:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:48:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_22.status 2024-09-13T09:48:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:48:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_22.status 2024-09-13T09:48:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-13T09:48:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:09.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=89 2024-09-13T09:48:09.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 161 = 89 2024-09-13T09:48:09.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:09.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:48:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_22.status 2024-09-13T09:48:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:48:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_22.status 2024-09-13T09:48:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-13T09:48:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:48:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:48:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:48:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:48:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:48:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 161 = '' 2024-09-13T09:48:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:48:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_22.status 2024-09-13T09:48:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:48:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_22.status 2024-09-13T09:48:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-13T09:48:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:10.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:48:10.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:48:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:48:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:48:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:48:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 161 = '' 2024-09-13T09:48:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:48:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_22.status 2024-09-13T09:48:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:48:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_22.status 2024-09-13T09:48:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-13T09:48:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:12.543 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-13T09:48:12.543 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:48:12.544 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-13T09:48:12.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-13T09:48:12.545 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-13T09:48:12.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-13T09:48:12.548 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2024-09-13T09:48:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=161 2024-09-13T09:48:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 161 = 161 2024-09-13T09:48:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-09-13T09:48:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:48:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:48:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:48:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:48:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:48:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-09-13T09:48:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_22': 2024-09-13T09:48:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:48:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:48:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:48:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 114ec53be41 2024-09-13T09:48:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.114ec53be41 2024-09-13T09:48:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:48:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:48:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:48:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:48:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:39:40 2024 2024-09-13T09:48:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:39:40 2024 2024-09-13T09:48:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:39:40 2024 2024-09-13T09:48:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:48:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:48:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:48:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 9cf71ada-0f10-4f47-91dd-9b7f8293279d 2024-09-13T09:48:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:48:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_22 snap 2024-09-13T09:48:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:48:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:48:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:48:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_22.export 2024-09-13T09:48:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_22.export 2024-09-13T09:48:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_22.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_22.export 2024-09-13T09:48:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_22.export 2024-09-13T09:48:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:48:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_22.export 2024-09-13T09:48:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:48:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_22.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_22.export 2024-09-13T09:48:14.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_22.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_22.export 2024-09-13T09:48:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:48:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_23 2024-09-13T09:48:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_23 snap 2024-09-13T09:48:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:48:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:48:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:48:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-09-13T09:48:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:48:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-09-13T09:48:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:48:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:48:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:48:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:48:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:48:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-09-13T09:48:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:48:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:48:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:48:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:48:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-09-13T09:48:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:48:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:48:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:48:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:48:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:48:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:48:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:48:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:48:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:48:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-09-13T09:48:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:48:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:48:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:48:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:48:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:48:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:48:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:48:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:48:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:48:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:48:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:48:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:48:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:48:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:48:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:48:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:48:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-09-13T09:48:16.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:48:16.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:48:16.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:48:16.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:48:16.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:48:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:48:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:48:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-09-13T09:48:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:48:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:48:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:48:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:48:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2024-09-13T09:48:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:48:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:48:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_23.status 2024-09-13T09:48:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_23.status 2024-09-13T09:48:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:48:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_23.status 2024-09-13T09:48:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_23.status 2024-09-13T09:48:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_23.status 2024-09-13T09:48:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:48:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_23.status 2024-09-13T09:48:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2024-09-13T09:48:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:48:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2024-09-13T09:48:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 163 2024-09-13T09:48:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_23.status 2024-09-13T09:48:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:48:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_23.status 2024-09-13T09:48:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2024-09-13T09:48:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:48:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=163 2024-09-13T09:48:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:48:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:48:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_23.status 2024-09-13T09:48:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:48:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_23.status 2024-09-13T09:48:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-13T09:48:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:48:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:17.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:48:17.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:48:17.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:48:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:48:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 163 = '' 2024-09-13T09:48:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:48:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_23.status 2024-09-13T09:48:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:48:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_23.status 2024-09-13T09:48:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-13T09:48:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:48:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:48:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:48:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:48:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:48:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 163 = '' 2024-09-13T09:48:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:48:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_23.status 2024-09-13T09:48:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:48:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_23.status 2024-09-13T09:48:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-13T09:48:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:19.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=163 2024-09-13T09:48:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 163 = 163 2024-09-13T09:48:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-09-13T09:48:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:48:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:48:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:48:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:48:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:48:20.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-09-13T09:48:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_23': 2024-09-13T09:48:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:48:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:48:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:48:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 128258664507 2024-09-13T09:48:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.128258664507 2024-09-13T09:48:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:48:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:48:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:48:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:48:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:39:53 2024 2024-09-13T09:48:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:39:53 2024 2024-09-13T09:48:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:39:53 2024 2024-09-13T09:48:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:48:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:48:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:48:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: b1f94b49-8135-472a-8920-462d2d90e565 2024-09-13T09:48:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:48:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_23 snap 2024-09-13T09:48:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:48:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:48:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:48:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_23.export 2024-09-13T09:48:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_23.export 2024-09-13T09:48:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_23.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_23.export 2024-09-13T09:48:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_23.export 2024-09-13T09:48:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:48:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_23.export 2024-09-13T09:48:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:48:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_23.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_23.export 2024-09-13T09:48:20.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_23.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_23.export 2024-09-13T09:48:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:48:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_24 2024-09-13T09:48:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_24 snap 2024-09-13T09:48:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:48:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:48:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:48:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-09-13T09:48:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:48:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-09-13T09:48:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:48:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:48:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:48:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:48:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:48:23.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-09-13T09:48:23.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:23.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:23.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:48:23.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:48:23.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:48:23.662 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:48:23.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-09-13T09:48:23.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:48:23.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:48:23.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:23.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:48:23.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:48:23.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:48:23.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:48:23.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:48:23.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:48:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:48:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-09-13T09:48:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:48:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:48:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:48:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:48:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:48:23.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:23.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:23.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:48:23.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:48:23.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:48:23.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:48:23.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:23.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:23.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:48:23.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:48:23.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:23.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:23.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:48:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:48:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:48:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:48:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:48:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-09-13T09:48:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-09-13T09:48:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:48:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:48:23.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:23.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2024-09-13T09:48:23.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:48:23.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:23.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:23.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:48:23.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:48:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:48:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:48:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:48:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:48:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:48:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:48:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:48:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:48:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:48:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:48:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:48:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:48:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:48:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:48:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:48:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:48:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2024-09-13T09:48:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-09-13T09:48:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:48:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:48:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2024-09-13T09:48:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:48:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:48:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:48:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:48:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:48:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:48:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:48:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:48:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:48:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:48:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:48:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:48:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:48:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:48:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:48:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:48:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:48:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:48:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:48:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2024-09-13T09:48:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-09-13T09:48:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:48:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:48:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_24 2024-09-13T09:48:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:48:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:48:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:48:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:48:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:48:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:48:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:48:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:48:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:48:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:48:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:48:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:23.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:23.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:48:23.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:48:23.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:23.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:48:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:48:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:48:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:48:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:48:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:48:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_24 2024-09-13T09:48:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:48:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:48:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:48:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:48:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:48:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:48:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:48:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-09-13T09:48:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:48:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:48:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:48:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:48:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2024-09-13T09:48:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:48:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:48:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_24.status 2024-09-13T09:48:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_24.status 2024-09-13T09:48:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:48:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_24.status 2024-09-13T09:48:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_24.status 2024-09-13T09:48:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_24.status 2024-09-13T09:48:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:48:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_24.status 2024-09-13T09:48:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2024-09-13T09:48:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:48:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2024-09-13T09:48:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 165 2024-09-13T09:48:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_24.status 2024-09-13T09:48:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:48:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_24.status 2024-09-13T09:48:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2024-09-13T09:48:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:48:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=165 2024-09-13T09:48:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:48:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:48:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_24.status 2024-09-13T09:48:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:48:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_24.status 2024-09-13T09:48:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-13T09:48:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:48:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:48:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:48:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:48:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:48:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 165 = '' 2024-09-13T09:48:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:48:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_24.status 2024-09-13T09:48:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:48:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_24.status 2024-09-13T09:48:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-13T09:48:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:48:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:48:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:48:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:48:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:48:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 165 = '' 2024-09-13T09:48:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:48:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_24.status 2024-09-13T09:48:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:48:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_24.status 2024-09-13T09:48:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-13T09:48:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=165 2024-09-13T09:48:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 165 = 165 2024-09-13T09:48:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-09-13T09:48:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:48:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:48:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:48:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:48:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:48:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-09-13T09:48:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_24': 2024-09-13T09:48:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:48:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:48:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:48:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 128bfdd69397 2024-09-13T09:48:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.128bfdd69397 2024-09-13T09:48:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:48:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:48:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:48:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:48:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:40:06 2024 2024-09-13T09:48:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:40:06 2024 2024-09-13T09:48:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:40:06 2024 2024-09-13T09:48:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:48:27.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:48:27.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:48:27.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: aa7749ba-cd93-4a24-906c-a357a64f71e8 2024-09-13T09:48:27.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:48:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_24 snap 2024-09-13T09:48:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:48:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:48:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:48:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_24.export 2024-09-13T09:48:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_24.export 2024-09-13T09:48:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_24.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_24.export 2024-09-13T09:48:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_24.export 2024-09-13T09:48:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:48:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_24.export 2024-09-13T09:48:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:48:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_24.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_24.export 2024-09-13T09:48:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_24.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_24.export 2024-09-13T09:48:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:48:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_25 2024-09-13T09:48:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_25 snap 2024-09-13T09:48:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:48:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:48:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:48:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-09-13T09:48:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:48:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-09-13T09:48:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:48:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:48:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:48:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:48:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:48:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-09-13T09:48:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:48:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:48:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:48:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:48:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-09-13T09:48:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:48:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:48:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:48:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:48:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:48:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:48:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:48:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:48:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:48:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-09-13T09:48:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:48:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:48:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:48:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:48:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:48:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:48:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:48:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:48:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:48:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:48:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:48:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:48:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:48:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:48:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:48:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:48:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-09-13T09:48:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-13T09:48:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:48:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:48:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2024-09-13T09:48:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:48:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:48:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:48:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:48:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:48:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:48:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:48:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:48:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:48:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:48:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:48:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:48:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:48:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:48:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:48:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:48:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:48:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:48:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:48:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2024-09-13T09:48:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-13T09:48:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:48:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:48:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_25 2024-09-13T09:48:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:48:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:48:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:48:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:48:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:48:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:48:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:48:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:48:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:48:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:48:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:48:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:48:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:48:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:48:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:48:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:48:30.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:48:30.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:48:30.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:48:30.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_25 2024-09-13T09:48:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-13T09:48:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:48:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:48:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_25 2024-09-13T09:48:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:48:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:48:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:48:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:48:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:48:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:48:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:48:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:48:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:48:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:48:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:48:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:48:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:48:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:48:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:48:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:48:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:48:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:48:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:48:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_25 2024-09-13T09:48:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-13T09:48:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:48:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:48:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:48:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 1 2024-09-13T09:48:32.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:48:32.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:48:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-09-13T09:48:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:48:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:48:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:48:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:48:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:48:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:48:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:48:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:48:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:48:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:48:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:48:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:48:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:48:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:32.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:48:32.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:48:32.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:48:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-09-13T09:48:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-13T09:48:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:48:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:48:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2024-09-13T09:48:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:48:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:48:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:48:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:48:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:48:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:48:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:48:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:48:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:48:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:48:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:48:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:48:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:48:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:48:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:48:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:48:32.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:48:32.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:48:32.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:48:32.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2024-09-13T09:48:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-13T09:48:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:48:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:48:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_25 2024-09-13T09:48:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:48:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:48:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:48:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:48:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:48:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:48:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:48:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:48:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:48:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:48:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:48:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:48:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:48:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:48:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:48:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:48:32.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:48:32.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:48:32.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:48:32.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_25 2024-09-13T09:48:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-13T09:48:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:48:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:48:32.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:32.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_25 2024-09-13T09:48:32.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:48:32.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:48:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:48:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:48:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:48:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:48:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:48:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:48:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:48:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:48:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:48:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:48:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:48:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:48:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:48:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:48:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:48:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:48:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:48:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_25 2024-09-13T09:48:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-13T09:48:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:48:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:48:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:48:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 2 2024-09-13T09:48:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:48:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:48:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-09-13T09:48:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:48:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:48:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:48:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:48:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:48:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:48:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:48:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:48:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:48:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:48:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:48:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:48:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:48:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:48:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:48:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:48:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-09-13T09:48:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-13T09:48:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:48:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:48:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2024-09-13T09:48:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:48:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:48:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:48:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:48:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:48:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:48:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:48:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:48:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:48:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:48:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:48:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:48:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:48:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:48:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:48:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:48:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:48:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:48:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:48:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2024-09-13T09:48:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-13T09:48:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:48:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:48:34.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:34.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_25 2024-09-13T09:48:34.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:48:34.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:34.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:48:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:48:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:48:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:48:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:48:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:48:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:48:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:48:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:48:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:48:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:48:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:48:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:48:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:48:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:48:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:48:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:48:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:48:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_25 2024-09-13T09:48:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:48:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:48:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:48:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:48:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:48:34.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:48:34.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:48:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-09-13T09:48:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:48:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:48:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:48:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:48:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2024-09-13T09:48:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:48:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:48:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_25.status 2024-09-13T09:48:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_25.status 2024-09-13T09:48:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:48:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_25.status 2024-09-13T09:48:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_25.status 2024-09-13T09:48:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_25.status 2024-09-13T09:48:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:48:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_25.status 2024-09-13T09:48:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2024-09-13T09:48:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:48:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2024-09-13T09:48:35.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 167 2024-09-13T09:48:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_25.status 2024-09-13T09:48:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:48:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_25.status 2024-09-13T09:48:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2024-09-13T09:48:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:48:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=167 2024-09-13T09:48:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:48:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:48:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_25.status 2024-09-13T09:48:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:48:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_25.status 2024-09-13T09:48:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-13T09:48:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:48:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:48:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:48:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:48:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:48:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 167 = '' 2024-09-13T09:48:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:48:36.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_25.status 2024-09-13T09:48:36.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:36.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:36.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:48:36.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_25.status 2024-09-13T09:48:36.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-13T09:48:36.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:48:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:48:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:48:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:48:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:48:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 167 = '' 2024-09-13T09:48:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:48:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_25.status 2024-09-13T09:48:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:48:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_25.status 2024-09-13T09:48:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-13T09:48:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=167 2024-09-13T09:48:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 167 = 167 2024-09-13T09:48:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-09-13T09:48:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:48:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:48:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:48:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:48:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:48:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-09-13T09:48:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_25': 2024-09-13T09:48:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:48:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:48:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:48:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c5704e438a 2024-09-13T09:48:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c5704e438a 2024-09-13T09:48:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:48:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:48:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:48:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:48:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:40:14 2024 2024-09-13T09:48:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:40:14 2024 2024-09-13T09:48:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:40:14 2024 2024-09-13T09:48:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:48:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:48:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:48:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: b4563f80-e68a-4c65-bea2-f31514c46d7c 2024-09-13T09:48:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:48:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_25 snap 2024-09-13T09:48:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:48:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:48:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:48:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_25.export 2024-09-13T09:48:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_25.export 2024-09-13T09:48:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_25.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_25.export 2024-09-13T09:48:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_25.export 2024-09-13T09:48:38.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:48:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_25.export 2024-09-13T09:48:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:48:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_25.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_25.export 2024-09-13T09:48:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_25.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_25.export 2024-09-13T09:48:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:48:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_26 2024-09-13T09:48:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_26 snap 2024-09-13T09:48:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:48:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:48:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:48:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-09-13T09:48:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:48:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-09-13T09:48:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:48:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:48:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:48:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:48:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:48:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-09-13T09:48:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:48:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:48:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:48:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:48:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-09-13T09:48:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:48:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:48:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:48:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:48:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:48:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:48:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:48:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:48:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:48:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-09-13T09:48:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:48:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:48:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:48:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:48:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:48:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:48:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:48:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:48:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:48:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:48:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:48:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:48:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:48:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:48:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:48:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:48:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-09-13T09:48:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-13T09:48:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:48:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:48:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-09-13T09:48:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:48:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:48:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:48:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:48:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:48:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:48:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:48:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:48:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:48:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:48:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:48:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:48:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:48:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:48:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:48:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:48:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:48:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:48:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:48:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-09-13T09:48:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-13T09:48:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:48:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:48:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2024-09-13T09:48:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:48:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:48:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:48:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:48:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:48:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:48:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:48:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:48:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:48:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:48:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:48:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:48:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:48:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:48:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:48:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:48:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:48:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:48:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:48:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2024-09-13T09:48:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:48:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:48:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:48:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:48:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:48:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:48:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:48:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-09-13T09:48:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:48:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:48:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:48:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:48:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2024-09-13T09:48:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:48:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:48:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_26.status 2024-09-13T09:48:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_26.status 2024-09-13T09:48:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:48:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_26.status 2024-09-13T09:48:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_26.status 2024-09-13T09:48:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_26.status 2024-09-13T09:48:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:48:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_26.status 2024-09-13T09:48:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2024-09-13T09:48:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:48:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2024-09-13T09:48:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 169 2024-09-13T09:48:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_26.status 2024-09-13T09:48:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:48:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_26.status 2024-09-13T09:48:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2024-09-13T09:48:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:42.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:48:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=169 2024-09-13T09:48:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:48:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:48:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_26.status 2024-09-13T09:48:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:48:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_26.status 2024-09-13T09:48:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-13T09:48:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=93 2024-09-13T09:48:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 169 = 93 2024-09-13T09:48:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:48:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_26.status 2024-09-13T09:48:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:48:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_26.status 2024-09-13T09:48:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-13T09:48:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=93 2024-09-13T09:48:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 169 = 93 2024-09-13T09:48:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:48:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_26.status 2024-09-13T09:48:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:48:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_26.status 2024-09-13T09:48:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-13T09:48:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:48:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:48:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:48:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:48:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:48:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 169 = '' 2024-09-13T09:48:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:48:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_26.status 2024-09-13T09:48:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:48:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_26.status 2024-09-13T09:48:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-13T09:48:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=169 2024-09-13T09:48:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 169 = 169 2024-09-13T09:48:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-09-13T09:48:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:48:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:48:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:48:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:48:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:48:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-09-13T09:48:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_26': 2024-09-13T09:48:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:48:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:48:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:48:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 129377a38553 2024-09-13T09:48:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.129377a38553 2024-09-13T09:48:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:48:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:48:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:48:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:48:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:40:22 2024 2024-09-13T09:48:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:40:22 2024 2024-09-13T09:48:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:40:22 2024 2024-09-13T09:48:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:48:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:48:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:48:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 9c358941-453b-4f18-9eae-0b0fdf040056 2024-09-13T09:48:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:48:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_26 snap 2024-09-13T09:48:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:48:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:48:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:48:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_26.export 2024-09-13T09:48:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_26.export 2024-09-13T09:48:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_26.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_26.export 2024-09-13T09:48:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_26.export 2024-09-13T09:48:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:48:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_26.export 2024-09-13T09:48:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:48:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_26.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_26.export 2024-09-13T09:48:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_26.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_26.export 2024-09-13T09:48:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:48:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_27 2024-09-13T09:48:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_27 snap 2024-09-13T09:48:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:48:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:48:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:48:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-09-13T09:48:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:48:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-09-13T09:48:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:48:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:48:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:48:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:48:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:48:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-09-13T09:48:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:48:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:48:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:48:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:48:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-09-13T09:48:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:48:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:48:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:48:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:48:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:48:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:48:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:48:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:48:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:48:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-09-13T09:48:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:48:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:48:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:48:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:48:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:48:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:48:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:48:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:48:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:48:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:48:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:48:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:48:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:48:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:48:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:48:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:48:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-09-13T09:48:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:48:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:48:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:48:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:48:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:48:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:48:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:48:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-09-13T09:48:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:48:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:48:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:48:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:48:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2024-09-13T09:48:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:48:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:48:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_27.status 2024-09-13T09:48:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_27.status 2024-09-13T09:48:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:48:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_27.status 2024-09-13T09:48:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_27.status 2024-09-13T09:48:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_27.status 2024-09-13T09:48:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:48:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_27.status 2024-09-13T09:48:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2024-09-13T09:48:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:48:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2024-09-13T09:48:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 171 2024-09-13T09:48:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_27.status 2024-09-13T09:48:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:48:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_27.status 2024-09-13T09:48:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2024-09-13T09:48:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:48:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=171 2024-09-13T09:48:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:48:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:48:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_27.status 2024-09-13T09:48:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:48:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_27.status 2024-09-13T09:48:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-13T09:48:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=94 2024-09-13T09:48:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 171 = 94 2024-09-13T09:48:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:48:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_27.status 2024-09-13T09:48:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:48:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_27.status 2024-09-13T09:48:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-13T09:48:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:51.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=94 2024-09-13T09:48:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 171 = 94 2024-09-13T09:48:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:48:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_27.status 2024-09-13T09:48:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:48:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_27.status 2024-09-13T09:48:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-13T09:48:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:48:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:48:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:48:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:48:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:48:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 171 = '' 2024-09-13T09:48:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:48:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_27.status 2024-09-13T09:48:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:48:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_27.status 2024-09-13T09:48:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-13T09:48:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=171 2024-09-13T09:48:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 171 = 171 2024-09-13T09:48:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-09-13T09:48:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:48:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:48:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:48:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:48:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:48:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-09-13T09:48:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_27': 2024-09-13T09:48:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:48:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:48:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:48:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1293a3b9c119 2024-09-13T09:48:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1293a3b9c119 2024-09-13T09:48:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:48:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:48:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:48:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:48:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:40:34 2024 2024-09-13T09:48:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:40:34 2024 2024-09-13T09:48:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:40:34 2024 2024-09-13T09:48:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:48:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:48:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:48:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 5ad6932e-f05d-4de5-89fb-186e6b5e26b1 2024-09-13T09:48:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:48:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_27 snap 2024-09-13T09:48:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:48:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:48:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:48:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_27.export 2024-09-13T09:48:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_27.export 2024-09-13T09:48:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_27.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_27.export 2024-09-13T09:48:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_27.export 2024-09-13T09:48:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:48:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_27.export 2024-09-13T09:48:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:48:55.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_27.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_27.export 2024-09-13T09:48:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_27.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_27.export 2024-09-13T09:48:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:48:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_28 2024-09-13T09:48:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_28 snap 2024-09-13T09:48:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:48:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:48:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:48:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-09-13T09:48:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:48:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-09-13T09:48:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:48:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:48:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:48:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:48:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:48:58.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-09-13T09:48:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:48:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:48:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:48:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:48:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-09-13T09:48:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:48:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:48:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:48:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:48:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:48:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:48:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:48:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:48:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:48:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-09-13T09:48:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:48:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:48:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:48:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:48:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:48:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:48:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:48:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:48:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:48:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:48:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:48:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:48:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:48:58.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:58.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:58.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:48:58.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:48:58.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:48:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:48:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-09-13T09:48:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-13T09:48:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:48:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:48:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2024-09-13T09:48:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:48:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:48:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:48:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:48:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:48:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:48:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:48:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:48:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:48:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:48:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:48:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:48:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2024-09-13T09:48:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-13T09:48:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:48:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:48:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_28 2024-09-13T09:48:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:48:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:48:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:48:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:48:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:48:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:48:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:48:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:48:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:48:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:48:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:48:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:48:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_28 2024-09-13T09:48:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-13T09:48:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:48:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:48:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:48:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_28 2024-09-13T09:48:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:48:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:48:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:48:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:48:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:48:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:48:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:48:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:48:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:48:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:48:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:48:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:48:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:48:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:48:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:48:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:48:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:48:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:48:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:48:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:48:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:48:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:48:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:48:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:48:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:48:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:48:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:48:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:48:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:48:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:48:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:48:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:48:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_28 2024-09-13T09:48:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:48:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:48:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:48:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:48:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:48:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:48:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:48:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:48:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2024-09-13T09:48:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:48:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2024-09-13T09:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_28.status 2024-09-13T09:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_28.status 2024-09-13T09:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_28.status 2024-09-13T09:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_28.status 2024-09-13T09:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_28.status 2024-09-13T09:48:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:48:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_28.status 2024-09-13T09:48:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2024-09-13T09:48:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:48:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2024-09-13T09:48:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 173 2024-09-13T09:48:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_28.status 2024-09-13T09:48:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:48:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:48:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_28.status 2024-09-13T09:48:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2024-09-13T09:48:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:48:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=173 2024-09-13T09:48:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:48:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:48:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_28.status 2024-09-13T09:48:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:48:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_28.status 2024-09-13T09:48:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-13T09:48:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=95 2024-09-13T09:48:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 173 = 95 2024-09-13T09:48:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:48:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_28.status 2024-09-13T09:48:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:48:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:48:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:48:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_28.status 2024-09-13T09:48:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-13T09:48:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:48:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:48:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:48:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:48:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:48:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:48:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:48:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 173 = '' 2024-09-13T09:48:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:48:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:49:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_28.status 2024-09-13T09:49:00.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:00.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:00.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:49:00.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_28.status 2024-09-13T09:49:00.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-13T09:49:00.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:49:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:49:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:49:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:49:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:49:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 173 = '' 2024-09-13T09:49:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:49:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_28.status 2024-09-13T09:49:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:49:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_28.status 2024-09-13T09:49:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-13T09:49:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:02.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=173 2024-09-13T09:49:02.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 173 = 173 2024-09-13T09:49:02.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:02.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-09-13T09:49:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:49:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:49:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:49:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:49:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-09-13T09:49:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_28': 2024-09-13T09:49:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:49:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:49:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:49:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1293b38dbc75 2024-09-13T09:49:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1293b38dbc75 2024-09-13T09:49:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:49:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:49:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:49:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:49:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:40:43 2024 2024-09-13T09:49:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:40:43 2024 2024-09-13T09:49:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:40:43 2024 2024-09-13T09:49:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:49:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:49:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:49:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: f0cfbdbd-fc4d-4303-a84d-1580b7778f89 2024-09-13T09:49:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:49:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_28 snap 2024-09-13T09:49:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:49:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:49:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_28.export 2024-09-13T09:49:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_28.export 2024-09-13T09:49:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_28.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_28.export 2024-09-13T09:49:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_28.export 2024-09-13T09:49:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:49:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_28.export 2024-09-13T09:49:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:49:03.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_28.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_28.export 2024-09-13T09:49:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_28.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_28.export 2024-09-13T09:49:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:49:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_29 2024-09-13T09:49:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_29 snap 2024-09-13T09:49:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:49:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:49:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-09-13T09:49:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:49:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-09-13T09:49:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:49:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:49:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:49:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:49:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:49:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-09-13T09:49:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:49:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:49:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:49:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:49:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-09-13T09:49:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:49:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:49:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:49:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:49:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:49:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:49:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:49:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:49:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:49:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-09-13T09:49:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:49:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:49:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:49:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:49:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:49:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:49:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:49:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:49:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:49:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:49:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:49:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:49:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:49:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:49:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:49:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:49:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-09-13T09:49:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-09-13T09:49:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:49:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:49:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2024-09-13T09:49:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:49:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:49:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:49:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:49:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:49:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:49:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:49:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:49:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:49:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:49:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:49:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:49:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:49:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:49:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:49:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:49:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2024-09-13T09:49:06.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-09-13T09:49:06.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:49:06.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:49:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2024-09-13T09:49:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:49:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:49:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:49:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:49:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:49:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:49:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:49:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:49:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:49:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:49:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:49:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:49:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:49:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:49:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:49:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:49:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:49:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:49:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:49:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2024-09-13T09:49:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:49:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:49:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:49:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:49:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:49:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:49:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:49:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2024-09-13T09:49:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:49:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:49:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:49:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:49:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2024-09-13T09:49:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:49:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:49:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_29.status 2024-09-13T09:49:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_29.status 2024-09-13T09:49:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:49:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_29.status 2024-09-13T09:49:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_29.status 2024-09-13T09:49:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_29.status 2024-09-13T09:49:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:49:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_29.status 2024-09-13T09:49:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2024-09-13T09:49:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:49:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2024-09-13T09:49:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 175 2024-09-13T09:49:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_29.status 2024-09-13T09:49:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:49:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_29.status 2024-09-13T09:49:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2024-09-13T09:49:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:49:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=175 2024-09-13T09:49:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:49:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:49:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_29.status 2024-09-13T09:49:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:49:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_29.status 2024-09-13T09:49:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-13T09:49:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=96 2024-09-13T09:49:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 175 = 96 2024-09-13T09:49:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:49:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_29.status 2024-09-13T09:49:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:49:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_29.status 2024-09-13T09:49:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-13T09:49:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:49:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:49:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:49:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:49:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:49:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 175 = '' 2024-09-13T09:49:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:49:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_29.status 2024-09-13T09:49:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:49:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_29.status 2024-09-13T09:49:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-13T09:49:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:08.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=175 2024-09-13T09:49:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 175 = 175 2024-09-13T09:49:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-09-13T09:49:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:49:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:49:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:49:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:49:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-09-13T09:49:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_29': 2024-09-13T09:49:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:49:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:49:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:49:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c58dcf9d4d 2024-09-13T09:49:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c58dcf9d4d 2024-09-13T09:49:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:49:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:49:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:49:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:49:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:40:49 2024 2024-09-13T09:49:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:40:49 2024 2024-09-13T09:49:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:40:49 2024 2024-09-13T09:49:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:49:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:49:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:49:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: fea3fd66-9c9a-4468-9fc6-b7638eeb26b7 2024-09-13T09:49:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:49:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_29 snap 2024-09-13T09:49:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:49:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:49:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_29.export 2024-09-13T09:49:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_29.export 2024-09-13T09:49:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_29.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_29.export 2024-09-13T09:49:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_29.export 2024-09-13T09:49:10.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:49:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_29.export 2024-09-13T09:49:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:49:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_29.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_29.export 2024-09-13T09:49:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_29.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_29.export 2024-09-13T09:49:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:49:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_30 2024-09-13T09:49:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_30 snap 2024-09-13T09:49:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:49:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:49:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-09-13T09:49:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:49:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-09-13T09:49:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:49:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:49:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:49:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:49:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:49:12.548 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7751709562252379) > weight (0.25) 2024-09-13T09:49:12.548 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.32721390104726866) > weight (0.25) 2024-09-13T09:49:12.549 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.49535740801575323) > weight (0.25) 2024-09-13T09:49:12.549 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.30502856126298916) > weight (0.25) 2024-09-13T09:49:12.549 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-09-13T09:49:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-09-13T09:49:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:49:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:49:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:49:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:49:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-09-13T09:49:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:49:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:49:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:49:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:49:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:49:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:49:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:49:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:49:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:49:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-09-13T09:49:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:49:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:49:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:49:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:49:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:49:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:49:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:49:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:49:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:49:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:49:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:49:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:49:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:49:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:49:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:49:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:49:13.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:13.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:13.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:13.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-09-13T09:49:13.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-09-13T09:49:13.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:49:13.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:49:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2024-09-13T09:49:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:49:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:49:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:49:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:49:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:49:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:49:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:49:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:49:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:49:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:49:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:49:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:49:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:49:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:49:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:49:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:49:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2024-09-13T09:49:13.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:49:13.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:49:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:49:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:49:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:49:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:49:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:49:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2024-09-13T09:49:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:49:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:49:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:49:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:49:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2024-09-13T09:49:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:49:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:49:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_30.status 2024-09-13T09:49:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_30.status 2024-09-13T09:49:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:49:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_30.status 2024-09-13T09:49:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_30.status 2024-09-13T09:49:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_30.status 2024-09-13T09:49:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:49:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_30.status 2024-09-13T09:49:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2024-09-13T09:49:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:49:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2024-09-13T09:49:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 177 2024-09-13T09:49:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_30.status 2024-09-13T09:49:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:49:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_30.status 2024-09-13T09:49:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2024-09-13T09:49:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:49:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=177 2024-09-13T09:49:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:49:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:49:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_30.status 2024-09-13T09:49:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:49:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_30.status 2024-09-13T09:49:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-13T09:49:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=97 2024-09-13T09:49:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 177 = 97 2024-09-13T09:49:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:49:14.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_30.status 2024-09-13T09:49:14.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:14.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:49:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_30.status 2024-09-13T09:49:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-13T09:49:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:49:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:49:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:49:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:49:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:49:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 177 = '' 2024-09-13T09:49:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:49:15.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_30.status 2024-09-13T09:49:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:49:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_30.status 2024-09-13T09:49:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-13T09:49:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=177 2024-09-13T09:49:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 177 = 177 2024-09-13T09:49:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-09-13T09:49:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:49:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:49:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:49:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:49:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-09-13T09:49:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_30': 2024-09-13T09:49:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:49:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:49:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:49:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12822bcdae2a 2024-09-13T09:49:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12822bcdae2a 2024-09-13T09:49:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:49:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:49:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:49:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:49:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:41:04 2024 2024-09-13T09:49:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:41:04 2024 2024-09-13T09:49:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:41:04 2024 2024-09-13T09:49:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:49:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:49:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:49:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 1b942165-b9f5-4c05-8666-d8d9900c56b6 2024-09-13T09:49:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:49:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_30 snap 2024-09-13T09:49:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:49:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:49:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_30.export 2024-09-13T09:49:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_30.export 2024-09-13T09:49:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_30.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_30.export 2024-09-13T09:49:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_30.export 2024-09-13T09:49:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:49:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_30.export 2024-09-13T09:49:17.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:49:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_30.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_30.export 2024-09-13T09:49:17.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_30.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_30.export 2024-09-13T09:49:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:49:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_31 2024-09-13T09:49:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_31 snap 2024-09-13T09:49:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:49:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:49:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-09-13T09:49:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:49:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-09-13T09:49:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:49:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:49:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:49:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:49:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:49:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-09-13T09:49:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:49:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:49:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:49:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:49:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-09-13T09:49:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:49:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:49:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:49:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:49:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:49:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:49:20.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:49:20.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:49:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:49:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-09-13T09:49:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:49:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:49:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:49:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:49:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:49:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:49:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:49:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:49:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:49:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:49:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:49:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:49:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:49:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:49:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:49:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:49:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-09-13T09:49:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-09-13T09:49:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:49:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:49:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2024-09-13T09:49:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:49:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:49:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:49:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:49:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:49:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:49:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:49:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:49:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:49:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:49:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:49:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:49:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:49:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:49:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:49:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:49:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2024-09-13T09:49:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:49:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:49:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:49:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:49:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:49:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:49:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:49:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2024-09-13T09:49:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:49:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:49:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:49:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:49:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2024-09-13T09:49:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:49:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:49:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_31.status 2024-09-13T09:49:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_31.status 2024-09-13T09:49:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:49:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_31.status 2024-09-13T09:49:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_31.status 2024-09-13T09:49:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_31.status 2024-09-13T09:49:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:49:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_31.status 2024-09-13T09:49:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2024-09-13T09:49:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:49:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2024-09-13T09:49:21.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 179 2024-09-13T09:49:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_31.status 2024-09-13T09:49:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:49:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_31.status 2024-09-13T09:49:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2024-09-13T09:49:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:49:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=179 2024-09-13T09:49:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:49:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:49:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_31.status 2024-09-13T09:49:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:49:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_31.status 2024-09-13T09:49:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-13T09:49:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:21.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=98 2024-09-13T09:49:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 179 = 98 2024-09-13T09:49:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:49:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_31.status 2024-09-13T09:49:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:49:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_31.status 2024-09-13T09:49:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-13T09:49:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:49:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:49:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:49:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:49:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:49:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 179 = '' 2024-09-13T09:49:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:49:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_31.status 2024-09-13T09:49:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:49:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_31.status 2024-09-13T09:49:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-13T09:49:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=179 2024-09-13T09:49:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 179 = 179 2024-09-13T09:49:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-09-13T09:49:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:49:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:49:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:49:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:49:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-09-13T09:49:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_31': 2024-09-13T09:49:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:49:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:49:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:49:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 128ba02e2c08 2024-09-13T09:49:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.128ba02e2c08 2024-09-13T09:49:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:49:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:49:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:49:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:49:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:41:17 2024 2024-09-13T09:49:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:41:17 2024 2024-09-13T09:49:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:41:17 2024 2024-09-13T09:49:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:49:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:49:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:49:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 1bb5e3e6-31d3-4cca-8cd7-40225052afae 2024-09-13T09:49:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:49:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_31 snap 2024-09-13T09:49:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:49:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:49:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_31.export 2024-09-13T09:49:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_31.export 2024-09-13T09:49:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_31.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_31.export 2024-09-13T09:49:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_31.export 2024-09-13T09:49:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:49:24.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_31.export 2024-09-13T09:49:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:49:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_31.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_31.export 2024-09-13T09:49:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_31.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_31.export 2024-09-13T09:49:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:49:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_32 2024-09-13T09:49:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_32 snap 2024-09-13T09:49:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:49:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:49:24.438 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4406935950898555) > weight (0.25) 2024-09-13T09:49:24.438 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8533622670209369) > weight (0.25) 2024-09-13T09:49:24.438 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6765079993467643) > weight (0.25) 2024-09-13T09:49:24.438 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-13T09:49:24.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-13T09:49:24.439 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-13T09:49:24.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:49:24.438+0000 7f9a51ae1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 36496) UID: 1000 2024-09-13T09:49:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-09-13T09:49:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:49:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-09-13T09:49:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:49:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:49:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:49:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:49:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:49:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-09-13T09:49:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:49:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:49:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:49:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-09-13T09:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:49:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:49:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:49:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-09-13T09:49:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:49:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:49:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:49:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:49:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:49:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:49:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:49:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:49:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:49:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:49:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:49:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:49:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:49:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:49:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:49:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:49:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-09-13T09:49:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-13T09:49:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:49:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:49:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2024-09-13T09:49:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:49:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:49:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:49:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:49:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:49:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:49:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:49:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:49:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:49:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:49:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:49:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:49:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:49:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:49:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:49:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:49:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2024-09-13T09:49:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-13T09:49:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:49:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:49:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_32 2024-09-13T09:49:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:49:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:49:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:49:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:49:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:49:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:49:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:49:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:49:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:49:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:49:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:49:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:49:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:49:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:49:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:49:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:49:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:49:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:49:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:49:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_32 2024-09-13T09:49:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:49:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:49:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:49:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:49:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:49:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:49:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:49:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2024-09-13T09:49:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:49:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:49:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:49:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:49:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2024-09-13T09:49:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:49:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:49:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_32.status 2024-09-13T09:49:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_32.status 2024-09-13T09:49:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:49:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_32.status 2024-09-13T09:49:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_32.status 2024-09-13T09:49:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_32.status 2024-09-13T09:49:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:49:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_32.status 2024-09-13T09:49:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2024-09-13T09:49:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:49:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2024-09-13T09:49:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 181 2024-09-13T09:49:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_32.status 2024-09-13T09:49:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:49:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_32.status 2024-09-13T09:49:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2024-09-13T09:49:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:49:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=181 2024-09-13T09:49:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:49:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:49:28.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_32.status 2024-09-13T09:49:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:49:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_32.status 2024-09-13T09:49:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-13T09:49:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=99 2024-09-13T09:49:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 181 = 99 2024-09-13T09:49:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:49:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_32.status 2024-09-13T09:49:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:49:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_32.status 2024-09-13T09:49:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-13T09:49:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:49:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:28.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:49:28.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:49:28.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:49:28.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:49:28.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 181 = '' 2024-09-13T09:49:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:49:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_32.status 2024-09-13T09:49:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:49:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_32.status 2024-09-13T09:49:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-13T09:49:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=181 2024-09-13T09:49:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 181 = 181 2024-09-13T09:49:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-09-13T09:49:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:49:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:49:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:49:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:49:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-09-13T09:49:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_32': 2024-09-13T09:49:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:49:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:49:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:49:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1293a33ebe9d 2024-09-13T09:49:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1293a33ebe9d 2024-09-13T09:49:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:49:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:49:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:49:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:49:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:41:25 2024 2024-09-13T09:49:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:41:25 2024 2024-09-13T09:49:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:41:25 2024 2024-09-13T09:49:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:49:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:49:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:49:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: ae63b672-de66-4994-b9e2-9cc620b8e9ef 2024-09-13T09:49:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:49:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_32 snap 2024-09-13T09:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_32.export 2024-09-13T09:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_32.export 2024-09-13T09:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_32.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_32.export 2024-09-13T09:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_32.export 2024-09-13T09:49:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:49:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_32.export 2024-09-13T09:49:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:49:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_32.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_32.export 2024-09-13T09:49:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_32.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_32.export 2024-09-13T09:49:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:49:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_33 2024-09-13T09:49:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_33 snap 2024-09-13T09:49:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:49:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:49:33.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-09-13T09:49:33.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:49:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-09-13T09:49:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:49:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:49:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:49:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:49:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:49:33.439 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-13T09:49:33.439 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:49:33.440 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-13T09:49:33.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-13T09:49:33.440 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-13T09:49:33.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-13T09:49:33.443 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2024-09-13T09:49:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-09-13T09:49:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:49:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:49:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:49:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:49:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-09-13T09:49:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:49:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:49:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:49:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:49:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:49:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:49:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:49:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:49:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:49:34.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:34.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-09-13T09:49:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:49:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:49:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:49:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:49:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:49:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:49:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:49:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:49:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:49:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:49:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:49:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:49:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:49:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:49:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:49:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:49:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-09-13T09:49:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-13T09:49:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:49:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:49:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-09-13T09:49:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:49:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:49:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:49:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:49:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:49:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:49:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:49:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:49:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:49:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:49:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:49:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:49:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:49:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:49:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:49:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:49:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-09-13T09:49:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-13T09:49:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:49:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:49:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-09-13T09:49:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:49:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:49:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:49:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:49:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:49:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:49:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:49:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:49:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:49:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:49:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:49:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:49:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:49:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:49:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:49:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:49:34.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:49:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:49:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:49:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-09-13T09:49:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:49:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:49:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:49:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:49:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:49:34.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:49:34.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:49:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2024-09-13T09:49:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:49:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:49:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:49:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:49:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2024-09-13T09:49:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:49:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:49:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_33.status 2024-09-13T09:49:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_33.status 2024-09-13T09:49:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:49:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_33.status 2024-09-13T09:49:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_33.status 2024-09-13T09:49:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_33.status 2024-09-13T09:49:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:49:34.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_33.status 2024-09-13T09:49:34.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2024-09-13T09:49:34.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:34.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:49:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2024-09-13T09:49:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 183 2024-09-13T09:49:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_33.status 2024-09-13T09:49:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:49:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_33.status 2024-09-13T09:49:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2024-09-13T09:49:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:49:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=183 2024-09-13T09:49:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:49:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:49:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_33.status 2024-09-13T09:49:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:49:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_33.status 2024-09-13T09:49:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-13T09:49:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=100 2024-09-13T09:49:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 183 = 100 2024-09-13T09:49:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:49:35.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_33.status 2024-09-13T09:49:35.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:49:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_33.status 2024-09-13T09:49:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-13T09:49:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:49:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:49:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:49:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:49:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:49:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 183 = '' 2024-09-13T09:49:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:49:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_33.status 2024-09-13T09:49:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:49:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_33.status 2024-09-13T09:49:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-13T09:49:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=183 2024-09-13T09:49:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 183 = 183 2024-09-13T09:49:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-09-13T09:49:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:49:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:49:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:49:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:49:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-09-13T09:49:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_33': 2024-09-13T09:49:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:49:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:49:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:49:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c589db8af9 2024-09-13T09:49:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c589db8af9 2024-09-13T09:49:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:49:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:49:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:49:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:49:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:41:33 2024 2024-09-13T09:49:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:41:33 2024 2024-09-13T09:49:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:41:33 2024 2024-09-13T09:49:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:49:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:49:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:49:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 7219b237-53fc-4501-baae-6c41f736571a 2024-09-13T09:49:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:49:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_33 snap 2024-09-13T09:49:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:49:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:49:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_33.export 2024-09-13T09:49:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_33.export 2024-09-13T09:49:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_33.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_33.export 2024-09-13T09:49:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_33.export 2024-09-13T09:49:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:49:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_33.export 2024-09-13T09:49:38.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:49:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_33.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_33.export 2024-09-13T09:49:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_33.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_33.export 2024-09-13T09:49:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:49:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_34 2024-09-13T09:49:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_34 snap 2024-09-13T09:49:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:49:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:49:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-09-13T09:49:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:49:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-09-13T09:49:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:49:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:49:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:49:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:49:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:49:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-09-13T09:49:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:49:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:49:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:49:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:49:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-09-13T09:49:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:49:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:49:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:49:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:49:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:49:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:49:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:49:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:49:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:49:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-09-13T09:49:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:49:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:49:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:49:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:49:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:49:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:49:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:49:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:49:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:49:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:40.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:49:40.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:49:40.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:49:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:49:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:49:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:49:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:49:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-09-13T09:49:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-13T09:49:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:49:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:49:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2024-09-13T09:49:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:49:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:49:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:49:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:49:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:49:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:49:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:49:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:49:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:49:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:49:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:49:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:49:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:49:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:49:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:49:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:49:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2024-09-13T09:49:40.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:49:40.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:49:40.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:49:40.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:49:40.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:49:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:49:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:49:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2024-09-13T09:49:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:49:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:49:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:49:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:49:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2024-09-13T09:49:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:49:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:49:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_34.status 2024-09-13T09:49:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_34.status 2024-09-13T09:49:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:49:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_34.status 2024-09-13T09:49:40.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_34.status 2024-09-13T09:49:40.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_34.status 2024-09-13T09:49:40.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:49:40.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_34.status 2024-09-13T09:49:40.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2024-09-13T09:49:40.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:40.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:40.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:49:40.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2024-09-13T09:49:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 185 2024-09-13T09:49:41.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_34.status 2024-09-13T09:49:41.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:41.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:41.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:49:41.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_34.status 2024-09-13T09:49:41.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2024-09-13T09:49:41.690 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:49:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=185 2024-09-13T09:49:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:49:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:49:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_34.status 2024-09-13T09:49:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:49:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_34.status 2024-09-13T09:49:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-13T09:49:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=101 2024-09-13T09:49:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 185 = 101 2024-09-13T09:49:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:49:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_34.status 2024-09-13T09:49:42.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:42.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:42.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:49:42.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_34.status 2024-09-13T09:49:42.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-13T09:49:42.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=101 2024-09-13T09:49:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 185 = 101 2024-09-13T09:49:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:49:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_34.status 2024-09-13T09:49:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:49:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_34.status 2024-09-13T09:49:43.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-13T09:49:43.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:49:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:49:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:49:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:49:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:49:43.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 185 = '' 2024-09-13T09:49:43.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:43.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:49:43.720 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2024-09-13T09:49:43.718+0000 7ff85b706640 -1 reset not still connected to 0x5568f9d12a90 2024-09-13T09:49:43.721 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2024-09-13T09:49:43.718+0000 7ff85b706640 -1 reset not still connected to 0x5569008d52b0 2024-09-13T09:49:43.721 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2024-09-13T09:49:43.718+0000 7ff85b706640 -1 reset not still connected to 0x556914671860 2024-09-13T09:49:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_34.status 2024-09-13T09:49:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:49:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_34.status 2024-09-13T09:49:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-13T09:49:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:45.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=185 2024-09-13T09:49:45.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 185 = 185 2024-09-13T09:49:45.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:45.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-09-13T09:49:45.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:45.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:45.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:49:45.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:45.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:49:45.137 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:49:45.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:49:46.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-09-13T09:49:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_34': 2024-09-13T09:49:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:49:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:49:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:49:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 128268934202 2024-09-13T09:49:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.128268934202 2024-09-13T09:49:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:49:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:49:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:49:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:49:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:41:44 2024 2024-09-13T09:49:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:41:44 2024 2024-09-13T09:49:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:41:44 2024 2024-09-13T09:49:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:49:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:49:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:49:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 7902759b-4c55-479e-b92d-2527b62dd9df 2024-09-13T09:49:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:49:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_34 snap 2024-09-13T09:49:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:49:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:49:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_34.export 2024-09-13T09:49:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_34.export 2024-09-13T09:49:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_34.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_34.export 2024-09-13T09:49:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_34.export 2024-09-13T09:49:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:49:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_34.export 2024-09-13T09:49:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:49:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_34.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_34.export 2024-09-13T09:49:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_34.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_34.export 2024-09-13T09:49:46.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:46.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:49:46.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_35 2024-09-13T09:49:46.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_35 snap 2024-09-13T09:49:46.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:46.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:46.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:49:46.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:46.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:49:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-09-13T09:49:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:49:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-09-13T09:49:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:49:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:49:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:49:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:49:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:49:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-09-13T09:49:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:49:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:49:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:49:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:49:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-09-13T09:49:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:49:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:49:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:49:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:49:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:49:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:49:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:49:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:49:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:49:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-09-13T09:49:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:49:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:49:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:49:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:49:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:49:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:49:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:49:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:49:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:49:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:49:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:49:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:49:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:49:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:49:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:49:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:49:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-09-13T09:49:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-09-13T09:49:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:49:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:49:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_35 2024-09-13T09:49:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:49:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:49:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:49:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:49:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:49:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:49:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:49:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:49:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:49:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:49:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:49:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:49:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:49:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:49:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:49:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:49:49.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:49.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:49.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:49.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_35 2024-09-13T09:49:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-09-13T09:49:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:49:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:49:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_35 2024-09-13T09:49:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:49:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:49:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:49:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:49:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:49:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:49:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:49:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:49:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:49:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:49:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:49:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:49:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:49:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:49:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:49:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:49:49.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:49:49.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:49:49.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:49:49.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_35 2024-09-13T09:49:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:49:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:49:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:49:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:49:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:49:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:49:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:49:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2024-09-13T09:49:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:49:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:49:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:49:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:49:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2024-09-13T09:49:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:49:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:49:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_35.status 2024-09-13T09:49:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_35.status 2024-09-13T09:49:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:49:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_35.status 2024-09-13T09:49:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_35.status 2024-09-13T09:49:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_35.status 2024-09-13T09:49:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:49:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_35.status 2024-09-13T09:49:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2024-09-13T09:49:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:49:50.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2024-09-13T09:49:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 187 2024-09-13T09:49:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_35.status 2024-09-13T09:49:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:49:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_35.status 2024-09-13T09:49:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2024-09-13T09:49:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:49:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=187 2024-09-13T09:49:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:49:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:49:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_35.status 2024-09-13T09:49:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:49:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_35.status 2024-09-13T09:49:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-13T09:49:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:49:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:49:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:49:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:49:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:49:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 187 = '' 2024-09-13T09:49:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:49:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_35.status 2024-09-13T09:49:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:49:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_35.status 2024-09-13T09:49:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-13T09:49:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:51.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:49:51.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:51.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:49:51.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:49:51.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:49:51.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:49:51.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 187 = '' 2024-09-13T09:49:51.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:51.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:49:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_35.status 2024-09-13T09:49:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:49:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_35.status 2024-09-13T09:49:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-13T09:49:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=187 2024-09-13T09:49:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 187 = 187 2024-09-13T09:49:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-09-13T09:49:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:49:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:49:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:49:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:49:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-09-13T09:49:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_35': 2024-09-13T09:49:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:49:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:49:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:49:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 128b98ac5dff 2024-09-13T09:49:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.128b98ac5dff 2024-09-13T09:49:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:49:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:49:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:49:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:49:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:41:50 2024 2024-09-13T09:49:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:41:50 2024 2024-09-13T09:49:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:41:50 2024 2024-09-13T09:49:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:49:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:49:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:49:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 6cbe51d3-590c-415c-bec2-78c533fae3f5 2024-09-13T09:49:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:49:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_35 snap 2024-09-13T09:49:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:49:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:49:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_35.export 2024-09-13T09:49:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_35.export 2024-09-13T09:49:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_35.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_35.export 2024-09-13T09:49:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_35.export 2024-09-13T09:49:53.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:49:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_35.export 2024-09-13T09:49:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:49:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_35.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_35.export 2024-09-13T09:49:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_35.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_35.export 2024-09-13T09:49:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:49:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_36 2024-09-13T09:49:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_36 snap 2024-09-13T09:49:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:49:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:49:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-09-13T09:49:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:49:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-09-13T09:49:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:49:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:49:55.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:49:55.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:49:55.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:49:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-09-13T09:49:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:49:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:49:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:49:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:49:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-09-13T09:49:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:49:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:49:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:49:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:49:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:49:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:49:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:49:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:49:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:49:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-09-13T09:49:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:49:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:49:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:49:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:49:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:49:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:49:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:49:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:49:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:49:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:49:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:49:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:49:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:49:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:49:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:49:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:49:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:49:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-09-13T09:49:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-13T09:49:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:49:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:49:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-09-13T09:49:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:49:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:49:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:49:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:49:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:49:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:49:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:49:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:49:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:49:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:49:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:49:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:49:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:49:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:49:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:49:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:49:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:49:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-09-13T09:49:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-13T09:49:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:49:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:49:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2024-09-13T09:49:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:49:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:49:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:49:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:49:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:49:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:49:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:49:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:49:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:49:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:49:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:49:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:49:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:49:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:49:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:49:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:49:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:49:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:49:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:49:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2024-09-13T09:49:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-13T09:49:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:49:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:49:57.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:49:57.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_36 2024-09-13T09:49:57.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:49:57.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:49:57.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:49:57.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:49:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:49:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:49:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:49:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:49:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:49:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:49:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:49:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:49:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:49:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:49:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:49:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:49:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:49:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:49:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:49:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:49:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:49:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:49:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:49:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:49:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:49:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:49:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:49:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:49:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:49:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:49:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:49:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:49:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_36 2024-09-13T09:49:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:49:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:49:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:49:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:49:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:49:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:49:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:49:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2024-09-13T09:49:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:49:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:49:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:49:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:49:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2024-09-13T09:49:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:49:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:49:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_36.status 2024-09-13T09:49:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_36.status 2024-09-13T09:49:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:49:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_36.status 2024-09-13T09:49:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_36.status 2024-09-13T09:49:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_36.status 2024-09-13T09:49:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:49:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_36.status 2024-09-13T09:49:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2024-09-13T09:49:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:49:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2024-09-13T09:49:57.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 189 2024-09-13T09:49:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_36.status 2024-09-13T09:49:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:49:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:49:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_36.status 2024-09-13T09:49:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2024-09-13T09:49:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:49:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=189 2024-09-13T09:49:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:49:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:49:57.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_36.status 2024-09-13T09:49:57.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:57.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:57.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:49:57.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_36.status 2024-09-13T09:49:57.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-13T09:49:57.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:58.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:49:58.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:49:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:49:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:49:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:49:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 189 = '' 2024-09-13T09:49:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:49:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_36.status 2024-09-13T09:49:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:49:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_36.status 2024-09-13T09:49:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-13T09:49:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:49:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:49:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:49:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:49:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:49:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 189 = '' 2024-09-13T09:49:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:49:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:49:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_36.status 2024-09-13T09:49:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:49:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_36.status 2024-09-13T09:49:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-13T09:49:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:49:59.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:49:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=189 2024-09-13T09:49:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 189 = 189 2024-09-13T09:49:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:49:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-09-13T09:49:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:49:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:49:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:49:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:49:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:49:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:49:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:50:00.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-09-13T09:50:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_36': 2024-09-13T09:50:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:50:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:50:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:50:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1293c5ae6852 2024-09-13T09:50:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1293c5ae6852 2024-09-13T09:50:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:50:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:50:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:50:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:50:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:41:57 2024 2024-09-13T09:50:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:41:57 2024 2024-09-13T09:50:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:41:57 2024 2024-09-13T09:50:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:50:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:50:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:50:00.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 9f803efc-9109-44ce-9b95-114b6dc86252 2024-09-13T09:50:00.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:50:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:50:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_36 snap 2024-09-13T09:50:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:50:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:50:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:50:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_36.export 2024-09-13T09:50:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_36.export 2024-09-13T09:50:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_36.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_36.export 2024-09-13T09:50:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_36.export 2024-09-13T09:50:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:50:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_36.export 2024-09-13T09:50:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:50:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_36.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_36.export 2024-09-13T09:50:00.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_36.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_36.export 2024-09-13T09:50:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:50:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:50:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_37 2024-09-13T09:50:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_37 snap 2024-09-13T09:50:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:50:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:50:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:50:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:50:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-09-13T09:50:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:50:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-09-13T09:50:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:50:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:50:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:50:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:50:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:50:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-09-13T09:50:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:50:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:50:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:50:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:50:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-09-13T09:50:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:50:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:50:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:50:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:50:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:50:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:50:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:50:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:50:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:50:03.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:03.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-09-13T09:50:03.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:50:03.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:03.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:50:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:50:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:50:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:50:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:50:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:50:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:50:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:50:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:50:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:50:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:50:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:50:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:50:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:50:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:50:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-09-13T09:50:03.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-09-13T09:50:03.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:03.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2024-09-13T09:50:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:50:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:50:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:50:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:50:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:50:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:50:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:50:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:50:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:50:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:50:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:50:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:50:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:50:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:50:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:50:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:50:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2024-09-13T09:50:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:50:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:50:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:50:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:50:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:50:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:50:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:50:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:50:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2024-09-13T09:50:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:50:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:50:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:50:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:50:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:50:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2024-09-13T09:50:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:50:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:50:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:50:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_37.status 2024-09-13T09:50:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_37.status 2024-09-13T09:50:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:50:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_37.status 2024-09-13T09:50:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_37.status 2024-09-13T09:50:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_37.status 2024-09-13T09:50:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:50:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_37.status 2024-09-13T09:50:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2024-09-13T09:50:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:50:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:50:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2024-09-13T09:50:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 191 2024-09-13T09:50:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_37.status 2024-09-13T09:50:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:50:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:50:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_37.status 2024-09-13T09:50:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2024-09-13T09:50:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:50:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:50:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=191 2024-09-13T09:50:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:50:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:50:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:50:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_37.status 2024-09-13T09:50:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:50:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_37.status 2024-09-13T09:50:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-13T09:50:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:50:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:50:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=104 2024-09-13T09:50:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 191 = 104 2024-09-13T09:50:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:50:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:50:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_37.status 2024-09-13T09:50:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:50:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_37.status 2024-09-13T09:50:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-13T09:50:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:50:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:50:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=104 2024-09-13T09:50:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 191 = 104 2024-09-13T09:50:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:50:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:50:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_37.status 2024-09-13T09:50:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:50:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_37.status 2024-09-13T09:50:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-13T09:50:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:50:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:50:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:50:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:50:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:50:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:50:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:50:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 191 = '' 2024-09-13T09:50:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:50:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:50:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_37.status 2024-09-13T09:50:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:50:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_37.status 2024-09-13T09:50:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-13T09:50:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:50:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:50:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=191 2024-09-13T09:50:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 191 = 191 2024-09-13T09:50:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:50:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-09-13T09:50:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:50:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:50:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:50:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:50:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:50:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-09-13T09:50:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_37': 2024-09-13T09:50:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c593317cc5 2024-09-13T09:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c593317cc5 2024-09-13T09:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:42:10 2024 2024-09-13T09:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:42:10 2024 2024-09-13T09:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:42:10 2024 2024-09-13T09:50:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:50:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:50:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:50:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 2428288c-879f-45c0-a7c5-4f2760c5f863 2024-09-13T09:50:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:50:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:50:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_37 snap 2024-09-13T09:50:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:50:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:50:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:50:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_37.export 2024-09-13T09:50:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_37.export 2024-09-13T09:50:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_37.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_37.export 2024-09-13T09:50:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_37.export 2024-09-13T09:50:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:50:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_37.export 2024-09-13T09:50:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:50:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_37.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_37.export 2024-09-13T09:50:09.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_37.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_37.export 2024-09-13T09:50:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:50:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:50:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_38 2024-09-13T09:50:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_38 snap 2024-09-13T09:50:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:50:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:50:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:50:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:50:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-09-13T09:50:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:50:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-09-13T09:50:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:50:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:50:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:50:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:50:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:50:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-09-13T09:50:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:50:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:50:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:50:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:50:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-09-13T09:50:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:50:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:50:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:50:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:50:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:50:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:50:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:50:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:50:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:50:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-09-13T09:50:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:50:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:50:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:50:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:50:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:50:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:50:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:50:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:50:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:50:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:50:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:50:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:50:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:50:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:50:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:50:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:50:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-09-13T09:50:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-13T09:50:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2024-09-13T09:50:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:50:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:50:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:50:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:50:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:50:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:50:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:50:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:50:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:50:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:50:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:50:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:50:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:50:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:50:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:50:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:50:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2024-09-13T09:50:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:50:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:50:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:50:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:50:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:50:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:50:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:50:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:50:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2024-09-13T09:50:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:50:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:50:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:50:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:50:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:50:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2024-09-13T09:50:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:50:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:50:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:50:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_38.status 2024-09-13T09:50:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_38.status 2024-09-13T09:50:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:50:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_38.status 2024-09-13T09:50:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_38.status 2024-09-13T09:50:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_38.status 2024-09-13T09:50:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:50:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_38.status 2024-09-13T09:50:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2024-09-13T09:50:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:50:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:50:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2024-09-13T09:50:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 193 2024-09-13T09:50:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_38.status 2024-09-13T09:50:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:50:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:50:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_38.status 2024-09-13T09:50:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2024-09-13T09:50:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:50:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:50:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=193 2024-09-13T09:50:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:50:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:50:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:50:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_38.status 2024-09-13T09:50:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:50:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_38.status 2024-09-13T09:50:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-13T09:50:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:50:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:50:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:50:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:50:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:50:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:50:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:50:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 193 = '' 2024-09-13T09:50:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:50:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:50:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_38.status 2024-09-13T09:50:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:50:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_38.status 2024-09-13T09:50:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-13T09:50:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:50:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:50:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:50:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:50:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:50:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:50:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:50:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 193 = '' 2024-09-13T09:50:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:50:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:50:14.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_38.status 2024-09-13T09:50:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:50:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_38.status 2024-09-13T09:50:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-13T09:50:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:50:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:50:14.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=193 2024-09-13T09:50:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 193 = 193 2024-09-13T09:50:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:50:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-09-13T09:50:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:50:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:50:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:50:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:50:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:50:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-09-13T09:50:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_38': 2024-09-13T09:50:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:50:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:50:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:50:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12821cb7d79f 2024-09-13T09:50:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12821cb7d79f 2024-09-13T09:50:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:50:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:50:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:50:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:50:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:42:20 2024 2024-09-13T09:50:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:42:20 2024 2024-09-13T09:50:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:42:20 2024 2024-09-13T09:50:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:50:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:50:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:50:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 53855d34-b4b5-4803-8724-2964e4fd7b9d 2024-09-13T09:50:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:50:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:50:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_38 snap 2024-09-13T09:50:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:50:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:50:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:50:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_38.export 2024-09-13T09:50:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_38.export 2024-09-13T09:50:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_38.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_38.export 2024-09-13T09:50:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_38.export 2024-09-13T09:50:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:50:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_38.export 2024-09-13T09:50:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:50:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_38.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_38.export 2024-09-13T09:50:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_38.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_38.export 2024-09-13T09:50:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:50:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:50:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_39 2024-09-13T09:50:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_39 snap 2024-09-13T09:50:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:50:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:50:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:50:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:50:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-09-13T09:50:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:50:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-09-13T09:50:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:50:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:50:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:50:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:50:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:50:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-09-13T09:50:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:50:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:50:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:50:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:50:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-09-13T09:50:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:50:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:50:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:50:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:50:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:50:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:50:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:50:18.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:50:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:50:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-09-13T09:50:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:50:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:50:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:50:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:50:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:50:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:50:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:50:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:50:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:50:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:50:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:50:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:50:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:50:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:50:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:50:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:50:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-09-13T09:50:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-13T09:50:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2024-09-13T09:50:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:50:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:50:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:50:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:50:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:50:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:50:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:50:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:50:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:50:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:50:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:50:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:50:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:50:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:50:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:50:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:50:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2024-09-13T09:50:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-13T09:50:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2024-09-13T09:50:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:50:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:50:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:50:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:50:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:50:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:50:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:50:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:50:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:50:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:50:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:50:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:50:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:50:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:50:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:50:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:50:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:50:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:50:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:50:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2024-09-13T09:50:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:50:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:50:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:50:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:50:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:50:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:50:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:50:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:50:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2024-09-13T09:50:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:50:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:50:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:50:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:50:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:50:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2024-09-13T09:50:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:50:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:50:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:50:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_39.status 2024-09-13T09:50:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_39.status 2024-09-13T09:50:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:50:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_39.status 2024-09-13T09:50:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_39.status 2024-09-13T09:50:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_39.status 2024-09-13T09:50:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:50:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_39.status 2024-09-13T09:50:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2024-09-13T09:50:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:50:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:50:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2024-09-13T09:50:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 195 2024-09-13T09:50:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_39.status 2024-09-13T09:50:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:50:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:50:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_39.status 2024-09-13T09:50:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2024-09-13T09:50:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:50:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:50:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=195 2024-09-13T09:50:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:50:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:50:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:50:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_39.status 2024-09-13T09:50:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:50:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_39.status 2024-09-13T09:50:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-13T09:50:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:50:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:50:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:50:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:50:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:50:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:50:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:50:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 195 = '' 2024-09-13T09:50:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:50:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:50:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_39.status 2024-09-13T09:50:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:50:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_39.status 2024-09-13T09:50:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-13T09:50:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:50:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:50:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:50:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:50:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:50:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:50:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:50:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 195 = '' 2024-09-13T09:50:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:50:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:50:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_39.status 2024-09-13T09:50:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:21.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:50:21.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_39.status 2024-09-13T09:50:21.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-13T09:50:21.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:50:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:50:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=195 2024-09-13T09:50:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 195 = 195 2024-09-13T09:50:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:50:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-09-13T09:50:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:50:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:50:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:50:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:50:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:50:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-09-13T09:50:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_39': 2024-09-13T09:50:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:50:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:50:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:50:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 128b526adc95 2024-09-13T09:50:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.128b526adc95 2024-09-13T09:50:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:50:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:50:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:50:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:50:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:42:29 2024 2024-09-13T09:50:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:42:29 2024 2024-09-13T09:50:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:42:29 2024 2024-09-13T09:50:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:50:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:50:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:50:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: f2864223-3140-4220-a2fa-798f1e72b93d 2024-09-13T09:50:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:50:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:50:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_39 snap 2024-09-13T09:50:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:50:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:50:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:50:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_39.export 2024-09-13T09:50:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_39.export 2024-09-13T09:50:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_39.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_39.export 2024-09-13T09:50:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_39.export 2024-09-13T09:50:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:50:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_39.export 2024-09-13T09:50:22.777 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:50:22.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_39.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_39.export 2024-09-13T09:50:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_39.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_39.export 2024-09-13T09:50:22.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:50:22.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:50:22.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_40 2024-09-13T09:50:22.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_40 snap 2024-09-13T09:50:22.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:50:22.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:22.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:50:22.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:50:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:50:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-09-13T09:50:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:50:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-09-13T09:50:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:50:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:50:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:50:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:50:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:50:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-09-13T09:50:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:50:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:50:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:50:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:50:25.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-09-13T09:50:25.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:50:25.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:50:25.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:25.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:50:25.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:50:25.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:50:25.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:50:25.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:50:25.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:50:25.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:50:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-09-13T09:50:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:50:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:50:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:50:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:50:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:50:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:50:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:50:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:50:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:50:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:50:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:50:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:50:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:50:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:50:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:50:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:50:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-09-13T09:50:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:50:25.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:50:25.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:50:25.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:50:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:50:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:50:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:50:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:50:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2024-09-13T09:50:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:50:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:50:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:50:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:50:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:50:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2024-09-13T09:50:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:50:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:50:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:50:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_40.status 2024-09-13T09:50:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_40.status 2024-09-13T09:50:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:50:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_40.status 2024-09-13T09:50:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_40.status 2024-09-13T09:50:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_40.status 2024-09-13T09:50:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:50:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_40.status 2024-09-13T09:50:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2024-09-13T09:50:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:50:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:50:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2024-09-13T09:50:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 197 2024-09-13T09:50:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_40.status 2024-09-13T09:50:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:50:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:50:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_40.status 2024-09-13T09:50:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2024-09-13T09:50:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:50:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:50:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=197 2024-09-13T09:50:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:50:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:50:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:50:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_40.status 2024-09-13T09:50:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:50:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_40.status 2024-09-13T09:50:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-13T09:50:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:50:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:50:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=107 2024-09-13T09:50:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 197 = 107 2024-09-13T09:50:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:50:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:50:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_40.status 2024-09-13T09:50:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:50:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_40.status 2024-09-13T09:50:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-13T09:50:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:50:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:50:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=107 2024-09-13T09:50:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 197 = 107 2024-09-13T09:50:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:50:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:50:27.550 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6902578276205795) > weight (0.25) 2024-09-13T09:50:27.550 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-13T09:50:27.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-13T09:50:27.551 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-13T09:50:27.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T09:50:27.550+0000 7f5f118da640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 55652) UID: 1000 2024-09-13T09:50:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_40.status 2024-09-13T09:50:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:50:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_40.status 2024-09-13T09:50:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-13T09:50:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:50:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:50:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:50:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:50:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:50:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:50:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:50:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 197 = '' 2024-09-13T09:50:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:50:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:50:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_40.status 2024-09-13T09:50:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:50:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_40.status 2024-09-13T09:50:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-13T09:50:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:50:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:50:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=197 2024-09-13T09:50:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 197 = 197 2024-09-13T09:50:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:50:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-09-13T09:50:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:50:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:50:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:50:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:50:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:50:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-09-13T09:50:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_40': 2024-09-13T09:50:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:50:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:50:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:50:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 129372de3f88 2024-09-13T09:50:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.129372de3f88 2024-09-13T09:50:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:50:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:50:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:50:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:50:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:42:40 2024 2024-09-13T09:50:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:42:40 2024 2024-09-13T09:50:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:42:40 2024 2024-09-13T09:50:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:50:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:50:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:50:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 80af752a-a561-44aa-a889-99b950f68127 2024-09-13T09:50:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:50:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:50:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_40 snap 2024-09-13T09:50:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:50:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:50:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:50:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_40.export 2024-09-13T09:50:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_40.export 2024-09-13T09:50:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_40.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_40.export 2024-09-13T09:50:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_40.export 2024-09-13T09:50:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:50:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_40.export 2024-09-13T09:50:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:50:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_40.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_40.export 2024-09-13T09:50:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_40.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_40.export 2024-09-13T09:50:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:50:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:50:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_41 2024-09-13T09:50:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_41 snap 2024-09-13T09:50:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:50:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:50:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:50:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:50:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-09-13T09:50:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:50:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-09-13T09:50:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:50:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:50:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:50:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:50:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:50:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-09-13T09:50:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:50:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:50:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:50:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:50:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:50:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:50:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-09-13T09:50:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:50:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:50:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:50:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:50:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:50:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:50:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:50:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:50:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:50:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-09-13T09:50:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:50:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:50:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:50:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:50:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:50:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:50:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:50:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:50:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:50:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:50:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:50:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:50:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:50:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:50:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:50:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:50:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:33.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:33.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-09-13T09:50:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2024-09-13T09:50:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:50:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:50:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:50:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:50:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:50:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:50:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:50:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:50:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:50:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:50:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:50:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:50:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:50:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:50:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:50:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:50:33.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:33.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:33.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:33.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2024-09-13T09:50:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-13T09:50:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2024-09-13T09:50:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:50:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:50:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:50:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:50:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:50:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:50:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:50:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:50:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:50:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:50:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:50:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:50:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:50:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:50:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:50:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:50:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:50:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:50:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:50:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2024-09-13T09:50:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_41 2024-09-13T09:50:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:50:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:50:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:50:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:50:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:50:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:50:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:50:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:50:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:50:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:50:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:50:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:50:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:50:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:50:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:50:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:50:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:50:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:50:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:50:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_41 2024-09-13T09:50:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:50:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 1 2024-09-13T09:50:34.551 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-13T09:50:34.551 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:50:34.552 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-13T09:50:34.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-13T09:50:34.553 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-13T09:50:34.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-13T09:50:34.555 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2024-09-13T09:50:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:50:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:50:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-09-13T09:50:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:50:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:50:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:50:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:50:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:50:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:50:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:50:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:50:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:50:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:50:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:50:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:50:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:50:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:50:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:50:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:50:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-09-13T09:50:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2024-09-13T09:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:50:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:50:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:50:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:50:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:50:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:50:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:50:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:50:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:50:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:50:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:50:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:50:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:50:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2024-09-13T09:50:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2024-09-13T09:50:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:50:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:50:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:50:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:50:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:50:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:50:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:50:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:50:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:50:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:50:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:50:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:50:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:50:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:50:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:50:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:50:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:50:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:50:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:50:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2024-09-13T09:50:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_41 2024-09-13T09:50:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:50:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:50:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:50:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:50:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:50:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:50:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:50:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:50:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:50:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:50:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:50:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:50:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:50:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:50:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:50:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:50:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:50:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:50:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:50:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_41 2024-09-13T09:50:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:50:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 2 2024-09-13T09:50:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:50:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:50:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-09-13T09:50:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:50:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:50:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:50:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:50:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:50:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:50:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:50:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:50:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:50:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:50:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:50:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:50:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:50:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:50:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:50:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:50:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-09-13T09:50:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2024-09-13T09:50:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:50:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:50:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:50:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:50:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:50:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:50:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:50:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:50:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:50:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:50:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:50:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:50:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:50:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:50:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:50:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:50:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2024-09-13T09:50:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2024-09-13T09:50:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:50:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:50:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:50:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:50:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:50:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:50:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:50:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:50:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:50:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:50:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:50:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:50:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:50:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:50:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:50:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:50:37.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:50:37.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:50:37.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:50:37.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2024-09-13T09:50:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_41 2024-09-13T09:50:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:50:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:50:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:50:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:50:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:50:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:50:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:50:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:50:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:50:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:50:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:50:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:50:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:50:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:37.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:50:37.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:50:37.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:50:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:50:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:50:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:50:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_41 2024-09-13T09:50:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:50:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 4 2024-09-13T09:50:39.444 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-13T09:50:39.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-13T09:50:39.444 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-13T09:50:39.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:50:39.444+0000 7f9e1f845640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 48828) UID: 1000 2024-09-13T09:50:41.445 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-13T09:50:41.446 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:50:41.446 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-13T09:50:41.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-13T09:50:41.446 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-13T09:50:41.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-13T09:50:41.449 INFO:tasks.rbd_mirror_thrash:waiting for 97 secs before thrashing 2024-09-13T09:50:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:50:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:50:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-09-13T09:50:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:50:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:50:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:50:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:50:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:50:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:50:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:50:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:50:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:50:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:50:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:50:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:50:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:50:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:50:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:50:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:50:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-09-13T09:50:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2024-09-13T09:50:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:50:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:50:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:50:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:50:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:50:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:50:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:50:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:50:42.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:50:42.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:42.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:42.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:50:42.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:50:42.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:42.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:42.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:42.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:50:42.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:50:42.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:42.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:42.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:50:42.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:50:42.078 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:50:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2024-09-13T09:50:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2024-09-13T09:50:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:50:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:50:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:50:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:50:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:50:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:50:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:50:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:50:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:50:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:50:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:50:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:50:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:50:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:50:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:50:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:50:42.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:50:42.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:50:42.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:50:42.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2024-09-13T09:50:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_41 2024-09-13T09:50:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:50:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:50:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:50:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:50:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:50:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:50:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:50:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:50:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:50:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:50:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:50:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:50:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:50:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:50:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:50:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:50:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:50:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:50:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:50:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_41 2024-09-13T09:50:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:50:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 8 2024-09-13T09:50:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:50:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:50:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-09-13T09:50:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:50:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:50:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:50:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:50:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:50:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:50:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:50:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:50:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:50:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:50:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:50:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:50:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:50:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:50:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:50:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:50:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-09-13T09:50:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:50.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:50.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2024-09-13T09:50:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:50:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:50:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:50:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:50.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:50.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:50.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:50:50.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:50:50.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:50.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:50.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:50:50.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:50:50.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:50:50.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:50:50.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:50.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:50.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:50:50.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:50:50.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:50.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:50.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:50.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:50:50.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:50:50.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:50.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:50.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:50:50.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:50:50.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:50:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2024-09-13T09:50:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2024-09-13T09:50:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:50:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:50:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:50:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:50:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:50:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:50:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:50:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:50:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:50:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:50:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:50:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:50:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:50:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:50:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:50:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:50:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:50:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:50:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:50:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2024-09-13T09:50:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_41 2024-09-13T09:50:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:50:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:50:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:50:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:50:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:50:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:50:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:50:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:50:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:50:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:50:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:50:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:50:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:50:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:50:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:50:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:50:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:50:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:50:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:50:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_41 2024-09-13T09:50:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:50:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 8 2024-09-13T09:50:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:50:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:50:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-09-13T09:50:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:50:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:50:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:50:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:50:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:50:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:50:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:50:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:50:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:50:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:50:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:50:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:50:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:50:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:50:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:50:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:50:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:50:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-09-13T09:50:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:58.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:58.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2024-09-13T09:50:58.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:50:58.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:58.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:58.904 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:50:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:50:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:50:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:50:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:50:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:50:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:50:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:50:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:50:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:50:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:50:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:50:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:50:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:50:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:50:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:50:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2024-09-13T09:50:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2024-09-13T09:50:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:50:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:50:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:50:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:50:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:50:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:50:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:50:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:50:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:50:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:50:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:50:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:50:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:50:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:50:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:50:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:50:59.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:50:59.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:50:59.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:50:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2024-09-13T09:50:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:50:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_41 2024-09-13T09:50:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:50:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:50:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:50:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:50:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:50:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:50:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:50:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:50:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:50:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:50:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:50:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:50:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:50:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:50:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:50:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:50:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:50:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:50:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:50:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:50:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:50:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:50:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:50:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:50:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:50:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:50:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:50:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:50:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:50:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:50:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:50:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:50:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_41 2024-09-13T09:50:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:50:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:50:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:50:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:50:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 8 2024-09-13T09:51:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:51:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:51:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-09-13T09:51:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:51:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:51:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:51:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:51:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:51:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:51:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:51:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:51:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:51:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:51:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:51:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:51:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:51:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:51:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:51:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:51:07.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:07.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:07.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:07.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-09-13T09:51:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-13T09:51:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:51:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:51:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2024-09-13T09:51:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:51:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:51:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:51:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:51:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:51:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:51:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:51:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:51:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:51:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:51:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:51:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:51:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:51:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:51:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:51:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:51:07.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:07.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:07.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:07.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2024-09-13T09:51:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:51:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:51:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:51:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:51:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:51:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:51:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:51:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2024-09-13T09:51:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:51:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:51:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:51:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:51:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2024-09-13T09:51:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:51:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:51:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_41.status 2024-09-13T09:51:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_41.status 2024-09-13T09:51:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:51:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_41.status 2024-09-13T09:51:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_41.status 2024-09-13T09:51:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_41.status 2024-09-13T09:51:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:51:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_41.status 2024-09-13T09:51:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2024-09-13T09:51:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:51:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2024-09-13T09:51:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 199 2024-09-13T09:51:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_41.status 2024-09-13T09:51:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:51:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_41.status 2024-09-13T09:51:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2024-09-13T09:51:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:51:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=199 2024-09-13T09:51:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:51:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:07.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:51:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_41.status 2024-09-13T09:51:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:51:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_41.status 2024-09-13T09:51:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-13T09:51:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:08.076 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=108 2024-09-13T09:51:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 199 = 108 2024-09-13T09:51:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:51:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_41.status 2024-09-13T09:51:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:51:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_41.status 2024-09-13T09:51:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-13T09:51:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=108 2024-09-13T09:51:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 199 = 108 2024-09-13T09:51:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:51:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_41.status 2024-09-13T09:51:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:51:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_41.status 2024-09-13T09:51:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-13T09:51:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:51:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:51:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:51:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:51:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:51:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 199 = '' 2024-09-13T09:51:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:51:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_41.status 2024-09-13T09:51:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:51:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_41.status 2024-09-13T09:51:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-13T09:51:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=199 2024-09-13T09:51:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 199 = 199 2024-09-13T09:51:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-09-13T09:51:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:51:11.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:51:11.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:51:11.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:51:11.036 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:51:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-09-13T09:51:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_41': 2024-09-13T09:51:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:51:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:51:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:51:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c56d120873 2024-09-13T09:51:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c56d120873 2024-09-13T09:51:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:51:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:51:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:51:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:51:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:42:53 2024 2024-09-13T09:51:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:42:53 2024 2024-09-13T09:51:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:42:53 2024 2024-09-13T09:51:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:51:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:51:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:51:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 80c5a437-de46-4aee-a1dd-7b12f06d8126 2024-09-13T09:51:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:51:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_41 snap 2024-09-13T09:51:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:51:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:51:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:51:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_41.export 2024-09-13T09:51:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_41.export 2024-09-13T09:51:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_41.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_41.export 2024-09-13T09:51:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_41.export 2024-09-13T09:51:12.251 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:51:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_41.export 2024-09-13T09:51:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:51:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_41.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_41.export 2024-09-13T09:51:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_41.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_41.export 2024-09-13T09:51:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:51:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_42 2024-09-13T09:51:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_42 snap 2024-09-13T09:51:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:51:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:51:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:51:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-09-13T09:51:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:51:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-09-13T09:51:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:51:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:51:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:51:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:51:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:51:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-09-13T09:51:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:51:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:51:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:51:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:51:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-09-13T09:51:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:51:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:51:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:51:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:51:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:51:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:51:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:51:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:51:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:51:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-09-13T09:51:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:51:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:51:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:51:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:51:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:51:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:51:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:51:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:51:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:51:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:51:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:51:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:51:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:51:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:51:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:51:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:51:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-09-13T09:51:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-13T09:51:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:51:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:51:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2024-09-13T09:51:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:51:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:51:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:51:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:51:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:51:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:51:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:51:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:51:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:51:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:51:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:51:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:51:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:51:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:51:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:51:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:51:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2024-09-13T09:51:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-13T09:51:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:51:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_42 2024-09-13T09:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:51:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:51:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:51:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:51:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:51:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:51:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:51:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:51:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:51:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:51:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:51:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:51:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:51:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:51:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:51:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_42 2024-09-13T09:51:15.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:51:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:51:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:51:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:51:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:51:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:51:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:51:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2024-09-13T09:51:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:51:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:51:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:51:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:51:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2024-09-13T09:51:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:51:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:51:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_42.status 2024-09-13T09:51:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_42.status 2024-09-13T09:51:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:51:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_42.status 2024-09-13T09:51:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_42.status 2024-09-13T09:51:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_42.status 2024-09-13T09:51:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:51:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_42.status 2024-09-13T09:51:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2024-09-13T09:51:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:51:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2024-09-13T09:51:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 201 2024-09-13T09:51:15.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_42.status 2024-09-13T09:51:15.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:15.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:15.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:51:15.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_42.status 2024-09-13T09:51:15.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2024-09-13T09:51:15.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:51:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=201 2024-09-13T09:51:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:51:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:51:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_42.status 2024-09-13T09:51:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:51:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_42.status 2024-09-13T09:51:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-13T09:51:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:16.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=109 2024-09-13T09:51:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 201 = 109 2024-09-13T09:51:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:51:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_42.status 2024-09-13T09:51:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:51:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_42.status 2024-09-13T09:51:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-13T09:51:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:51:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:51:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:51:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:51:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:51:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 201 = '' 2024-09-13T09:51:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:51:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_42.status 2024-09-13T09:51:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:51:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_42.status 2024-09-13T09:51:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-13T09:51:17.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:51:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:51:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:51:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:51:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:51:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 201 = '' 2024-09-13T09:51:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:51:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_42.status 2024-09-13T09:51:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:51:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_42.status 2024-09-13T09:51:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-13T09:51:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=201 2024-09-13T09:51:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 201 = 201 2024-09-13T09:51:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-09-13T09:51:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:51:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:51:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:51:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:51:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:51:20.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-09-13T09:51:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_42': 2024-09-13T09:51:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:51:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:51:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:51:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1282551c6cd6 2024-09-13T09:51:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1282551c6cd6 2024-09-13T09:51:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:51:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:51:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:51:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:51:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:43:03 2024 2024-09-13T09:51:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:43:03 2024 2024-09-13T09:51:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:43:03 2024 2024-09-13T09:51:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:51:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:51:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:51:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 66db570e-e124-4bcd-9c58-ccbd57311102 2024-09-13T09:51:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:51:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_42 snap 2024-09-13T09:51:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:51:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:51:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:51:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_42.export 2024-09-13T09:51:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_42.export 2024-09-13T09:51:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_42.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_42.export 2024-09-13T09:51:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_42.export 2024-09-13T09:51:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:51:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_42.export 2024-09-13T09:51:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:51:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_42.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_42.export 2024-09-13T09:51:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_42.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_42.export 2024-09-13T09:51:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:51:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_43 2024-09-13T09:51:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_43 snap 2024-09-13T09:51:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:51:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:51:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:51:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-09-13T09:51:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:51:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-09-13T09:51:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:51:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:51:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:51:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:51:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:51:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-09-13T09:51:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:51:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:51:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:51:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:51:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-09-13T09:51:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:51:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:51:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:51:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:51:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:51:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:51:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:51:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:51:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:51:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-09-13T09:51:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:51:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:51:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:51:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:51:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:51:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:51:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:51:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:51:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:51:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-09-13T09:51:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-09-13T09:51:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:51:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:51:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2024-09-13T09:51:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:51:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:51:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:51:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:51:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:51:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:51:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:51:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:51:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:51:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:51:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:51:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:51:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:51:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:51:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:51:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:51:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2024-09-13T09:51:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:51:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:51:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:51:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:51:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:51:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:51:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:51:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2024-09-13T09:51:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:51:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:51:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:51:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:51:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2024-09-13T09:51:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:51:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:51:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_43.status 2024-09-13T09:51:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_43.status 2024-09-13T09:51:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:51:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_43.status 2024-09-13T09:51:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_43.status 2024-09-13T09:51:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_43.status 2024-09-13T09:51:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:51:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_43.status 2024-09-13T09:51:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2024-09-13T09:51:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:51:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2024-09-13T09:51:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 203 2024-09-13T09:51:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_43.status 2024-09-13T09:51:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:51:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_43.status 2024-09-13T09:51:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2024-09-13T09:51:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:51:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=203 2024-09-13T09:51:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:51:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:51:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_43.status 2024-09-13T09:51:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:51:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_43.status 2024-09-13T09:51:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-13T09:51:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=110 2024-09-13T09:51:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 203 = 110 2024-09-13T09:51:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:51:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_43.status 2024-09-13T09:51:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:51:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_43.status 2024-09-13T09:51:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-13T09:51:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=110 2024-09-13T09:51:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 203 = 110 2024-09-13T09:51:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:51:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_43.status 2024-09-13T09:51:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:51:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_43.status 2024-09-13T09:51:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-13T09:51:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:51:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:51:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:51:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:51:26.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:51:26.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 203 = '' 2024-09-13T09:51:26.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:26.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:51:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_43.status 2024-09-13T09:51:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:51:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_43.status 2024-09-13T09:51:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-13T09:51:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=203 2024-09-13T09:51:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 203 = 203 2024-09-13T09:51:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-09-13T09:51:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:51:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:51:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:51:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:51:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:51:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-09-13T09:51:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_43': 2024-09-13T09:51:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:51:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:51:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:51:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 128b6e5dfda4 2024-09-13T09:51:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.128b6e5dfda4 2024-09-13T09:51:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:51:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:51:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:51:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:51:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:43:16 2024 2024-09-13T09:51:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:43:16 2024 2024-09-13T09:51:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:43:16 2024 2024-09-13T09:51:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:51:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:51:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:51:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 69d778b5-940c-4aa1-8021-ea2d132d002e 2024-09-13T09:51:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:51:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_43 snap 2024-09-13T09:51:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:51:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:51:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:51:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_43.export 2024-09-13T09:51:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_43.export 2024-09-13T09:51:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_43.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_43.export 2024-09-13T09:51:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_43.export 2024-09-13T09:51:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:51:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_43.export 2024-09-13T09:51:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:51:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_43.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_43.export 2024-09-13T09:51:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_43.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_43.export 2024-09-13T09:51:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:51:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_44 2024-09-13T09:51:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_44 snap 2024-09-13T09:51:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:51:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:51:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:51:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-09-13T09:51:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:51:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-09-13T09:51:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:51:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:51:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:51:30.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:51:30.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:51:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-09-13T09:51:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:51:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:51:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:51:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:51:31.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-09-13T09:51:31.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:51:31.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:31.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:51:31.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:51:31.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:51:31.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:51:31.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:51:31.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:51:31.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:51:31.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:51:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-09-13T09:51:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:51:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:51:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:51:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:51:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:51:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:51:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:51:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:51:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:51:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:51:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:51:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:51:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:51:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:51:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:51:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:51:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-09-13T09:51:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-13T09:51:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:51:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:51:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2024-09-13T09:51:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:51:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:51:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:51:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:51:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:51:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:51:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:51:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:51:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:51:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:51:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:51:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:51:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:51:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:51:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:51:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:51:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2024-09-13T09:51:32.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:51:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:51:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:51:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:51:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:51:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:51:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:51:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2024-09-13T09:51:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:51:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:51:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:51:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:51:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2024-09-13T09:51:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:51:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:51:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_44.status 2024-09-13T09:51:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_44.status 2024-09-13T09:51:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:51:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_44.status 2024-09-13T09:51:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_44.status 2024-09-13T09:51:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_44.status 2024-09-13T09:51:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:51:32.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_44.status 2024-09-13T09:51:32.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2024-09-13T09:51:32.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:32.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:32.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:51:32.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2024-09-13T09:51:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 205 2024-09-13T09:51:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_44.status 2024-09-13T09:51:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:51:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_44.status 2024-09-13T09:51:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2024-09-13T09:51:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:51:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=205 2024-09-13T09:51:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:51:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:51:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_44.status 2024-09-13T09:51:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:51:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_44.status 2024-09-13T09:51:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-13T09:51:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=111 2024-09-13T09:51:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 205 = 111 2024-09-13T09:51:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:51:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_44.status 2024-09-13T09:51:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:51:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_44.status 2024-09-13T09:51:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-13T09:51:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=111 2024-09-13T09:51:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 205 = 111 2024-09-13T09:51:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:51:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_44.status 2024-09-13T09:51:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:51:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_44.status 2024-09-13T09:51:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-13T09:51:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:51:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:51:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:51:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:51:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:51:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 205 = '' 2024-09-13T09:51:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:51:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_44.status 2024-09-13T09:51:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:51:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_44.status 2024-09-13T09:51:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-13T09:51:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:36.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=205 2024-09-13T09:51:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 205 = 205 2024-09-13T09:51:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-09-13T09:51:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:51:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:51:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:51:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:51:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:51:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-09-13T09:51:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_44': 2024-09-13T09:51:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:51:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:51:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:51:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12932a80af83 2024-09-13T09:51:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12932a80af83 2024-09-13T09:51:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:51:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:51:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:51:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:51:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:43:26 2024 2024-09-13T09:51:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:43:26 2024 2024-09-13T09:51:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:43:26 2024 2024-09-13T09:51:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:51:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:51:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:51:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 90e5e796-ce61-474c-b054-7f1a077502a8 2024-09-13T09:51:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:51:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_44 snap 2024-09-13T09:51:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:51:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:51:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:51:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_44.export 2024-09-13T09:51:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_44.export 2024-09-13T09:51:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_44.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_44.export 2024-09-13T09:51:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_44.export 2024-09-13T09:51:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:51:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_44.export 2024-09-13T09:51:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:51:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_44.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_44.export 2024-09-13T09:51:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_44.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_44.export 2024-09-13T09:51:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:51:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_45 2024-09-13T09:51:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_45 snap 2024-09-13T09:51:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:51:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:51:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:51:39.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-09-13T09:51:39.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:39.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:39.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:51:39.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-09-13T09:51:39.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:39.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:39.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:51:39.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:51:39.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:51:39.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:51:39.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:51:40.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-09-13T09:51:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:51:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:51:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:51:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:51:40.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-09-13T09:51:40.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:51:40.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:51:40.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:40.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:51:40.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:51:40.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:51:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:51:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:51:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:51:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:51:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-09-13T09:51:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:51:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:51:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:51:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:51:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:51:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:51:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:51:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:51:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:51:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:51:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:51:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:51:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:51:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:51:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:51:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:51:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-09-13T09:51:40.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-13T09:51:40.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:51:40.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:51:40.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:40.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2024-09-13T09:51:40.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:51:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:51:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:51:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:51:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:51:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:51:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:51:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:51:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:51:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:51:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:51:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:51:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:51:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:51:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:51:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:51:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2024-09-13T09:51:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:51:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:51:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:51:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:51:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:51:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:51:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:51:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2024-09-13T09:51:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:51:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:51:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:51:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:51:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2024-09-13T09:51:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:51:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:51:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_45.status 2024-09-13T09:51:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_45.status 2024-09-13T09:51:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:51:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_45.status 2024-09-13T09:51:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_45.status 2024-09-13T09:51:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_45.status 2024-09-13T09:51:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:51:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_45.status 2024-09-13T09:51:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2024-09-13T09:51:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:51:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2024-09-13T09:51:41.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 207 2024-09-13T09:51:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_45.status 2024-09-13T09:51:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:51:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_45.status 2024-09-13T09:51:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2024-09-13T09:51:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:51:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=207 2024-09-13T09:51:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:51:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:51:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_45.status 2024-09-13T09:51:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:51:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_45.status 2024-09-13T09:51:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-13T09:51:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=112 2024-09-13T09:51:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 207 = 112 2024-09-13T09:51:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:51:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_45.status 2024-09-13T09:51:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:51:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_45.status 2024-09-13T09:51:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-13T09:51:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:51:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:51:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:51:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:51:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:51:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 207 = '' 2024-09-13T09:51:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:51:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_45.status 2024-09-13T09:51:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:51:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_45.status 2024-09-13T09:51:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-13T09:51:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:51:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:51:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:51:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:51:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:51:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 207 = '' 2024-09-13T09:51:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:51:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_45.status 2024-09-13T09:51:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:51:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_45.status 2024-09-13T09:51:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-13T09:51:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=207 2024-09-13T09:51:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 207 = 207 2024-09-13T09:51:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-09-13T09:51:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:51:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:51:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:51:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:51:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:51:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-09-13T09:51:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_45': 2024-09-13T09:51:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:51:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:51:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:51:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c5d1721a36 2024-09-13T09:51:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c5d1721a36 2024-09-13T09:51:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:51:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:51:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:51:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:51:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:43:35 2024 2024-09-13T09:51:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:43:35 2024 2024-09-13T09:51:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:43:35 2024 2024-09-13T09:51:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:51:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:51:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:51:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 24eb5df8-5de6-4058-952e-856c1cb902e7 2024-09-13T09:51:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:51:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_45 snap 2024-09-13T09:51:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:51:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:51:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:51:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_45.export 2024-09-13T09:51:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_45.export 2024-09-13T09:51:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_45.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_45.export 2024-09-13T09:51:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_45.export 2024-09-13T09:51:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:51:45.719 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_45.export 2024-09-13T09:51:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:51:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_45.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_45.export 2024-09-13T09:51:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_45.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_45.export 2024-09-13T09:51:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:51:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_46 2024-09-13T09:51:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_46 snap 2024-09-13T09:51:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:51:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:51:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:51:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-09-13T09:51:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:51:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-09-13T09:51:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:51:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:51:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:51:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:51:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:51:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-09-13T09:51:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:51:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:51:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:51:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:51:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-09-13T09:51:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:51:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:51:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:51:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:51:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:51:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:51:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:51:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:51:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:51:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-09-13T09:51:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:51:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:51:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:51:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:51:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:51:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:51:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:51:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:51:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:51:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:51:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:51:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:51:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:51:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:51:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:51:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:51:48.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:48.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:48.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:48.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-09-13T09:51:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-13T09:51:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:51:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:51:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2024-09-13T09:51:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:51:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:51:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:51:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:51:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:51:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:51:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:51:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:51:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:51:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:51:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:51:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:51:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:51:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:51:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:51:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:51:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2024-09-13T09:51:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-13T09:51:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:51:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:51:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_46 2024-09-13T09:51:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:51:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:51:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:51:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:51:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:51:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:51:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:51:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:51:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:51:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:51:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:51:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:51:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:51:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:51:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:51:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:51:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:51:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:51:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:51:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_46 2024-09-13T09:51:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:51:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:51:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:51:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:51:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:51:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:51:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:51:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2024-09-13T09:51:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:51:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:51:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:51:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:51:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2024-09-13T09:51:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:51:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:51:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_46.status 2024-09-13T09:51:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_46.status 2024-09-13T09:51:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:51:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_46.status 2024-09-13T09:51:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_46.status 2024-09-13T09:51:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_46.status 2024-09-13T09:51:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:51:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_46.status 2024-09-13T09:51:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2024-09-13T09:51:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:51:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2024-09-13T09:51:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 209 2024-09-13T09:51:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_46.status 2024-09-13T09:51:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:51:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_46.status 2024-09-13T09:51:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2024-09-13T09:51:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:51:49.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=209 2024-09-13T09:51:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:51:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:51:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_46.status 2024-09-13T09:51:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:51:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_46.status 2024-09-13T09:51:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-13T09:51:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=113 2024-09-13T09:51:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 209 = 113 2024-09-13T09:51:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:51:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_46.status 2024-09-13T09:51:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:51:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_46.status 2024-09-13T09:51:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-13T09:51:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:51:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:51:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:51:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:51:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:51:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 209 = '' 2024-09-13T09:51:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:51:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_46.status 2024-09-13T09:51:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:51:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_46.status 2024-09-13T09:51:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-13T09:51:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:51:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:51:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:51:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:51:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:51:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 209 = '' 2024-09-13T09:51:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:51:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:51:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_46.status 2024-09-13T09:51:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:51:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_46.status 2024-09-13T09:51:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-13T09:51:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:51:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:51:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=209 2024-09-13T09:51:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 209 = 209 2024-09-13T09:51:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-09-13T09:51:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:51:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:51:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:51:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:51:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:51:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-09-13T09:51:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_46': 2024-09-13T09:51:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:51:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:51:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:51:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12acdb389acc 2024-09-13T09:51:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12acdb389acc 2024-09-13T09:51:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:51:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:51:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:51:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:51:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:43:46 2024 2024-09-13T09:51:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:43:46 2024 2024-09-13T09:51:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:43:46 2024 2024-09-13T09:51:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:51:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:51:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:51:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: b2303ec4-1348-4438-91f9-318b0bbb03c1 2024-09-13T09:51:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:51:53.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_46 snap 2024-09-13T09:51:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:51:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:51:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:51:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_46.export 2024-09-13T09:51:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_46.export 2024-09-13T09:51:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_46.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_46.export 2024-09-13T09:51:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_46.export 2024-09-13T09:51:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:51:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_46.export 2024-09-13T09:51:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:51:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_46.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_46.export 2024-09-13T09:51:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_46.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_46.export 2024-09-13T09:51:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:51:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:51:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_47 2024-09-13T09:51:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_47 snap 2024-09-13T09:51:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:51:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:51:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:51:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:51:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-09-13T09:51:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:51:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-09-13T09:51:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:51:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:51:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:51:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:51:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:51:55.556 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5796017797315578) > weight (0.25) 2024-09-13T09:51:55.557 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8118062362433661) > weight (0.25) 2024-09-13T09:51:55.557 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8155834736882852) > weight (0.25) 2024-09-13T09:51:55.557 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-13T09:51:55.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-13T09:51:55.558 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-13T09:51:55.558 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-13T09:51:55.558 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:51:55.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T09:51:55.557+0000 7f9d47128640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 36342) UID: 1000 2024-09-13T09:51:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-09-13T09:51:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:51:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:51:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:51:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:51:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:51:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:51:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-09-13T09:51:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:51:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:51:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:51:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:51:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:51:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:51:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:51:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:51:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:51:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-09-13T09:51:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:51:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:51:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:51:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:51:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:51:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:51:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:51:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:51:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:51:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:51:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:51:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:51:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:51:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:51:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:51:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:51:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-09-13T09:51:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:51:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:51:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:51:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2024-09-13T09:51:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:51:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:51:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:51:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:51:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:51:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:51:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:51:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:51:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:51:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:51:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:51:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:51:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:51:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:51:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:51:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:51:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2024-09-13T09:51:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:51:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:51:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:51:56.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:56.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2024-09-13T09:51:56.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:51:56.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:51:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:51:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:51:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:51:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:51:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:51:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:51:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:51:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:56.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:56.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:51:56.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:51:56.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:56.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:56.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:56.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:51:56.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:51:56.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:56.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:56.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:51:56.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:51:56.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:51:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:51:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:51:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:51:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2024-09-13T09:51:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:51:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:51:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:51:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2024-09-13T09:51:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:51:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:51:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:51:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:51:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:51:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:51:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:51:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:51:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:51:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:51:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:51:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:51:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:51:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:51:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:51:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:51:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:51:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:51:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:51:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2024-09-13T09:51:56.639 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-13T09:51:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:51:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 1 2024-09-13T09:51:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:51:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:51:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-09-13T09:51:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:51:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:51:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:51:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:51:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:51:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:51:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:51:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:51:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:51:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:51:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:51:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:51:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:51:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:51:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:51:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:51:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:51:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-09-13T09:51:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:51:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:51:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:51:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2024-09-13T09:51:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:51:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:51:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:51:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:51:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:51:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:51:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:51:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:51:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:51:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:51:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:51:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:57.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:51:57.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:51:57.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:57.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:57.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:51:57.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:51:57.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:51:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:51:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2024-09-13T09:51:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:51:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:51:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:51:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2024-09-13T09:51:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:51:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:51:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:51:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:51:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:51:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:51:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:51:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:51:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:51:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:51:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:51:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:51:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:51:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:51:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:51:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:51:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:51:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:51:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:51:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2024-09-13T09:51:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:51:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:51:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:51:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:51:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2024-09-13T09:51:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:51:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:51:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:51:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:51:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:51:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:51:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:51:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:51:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:51:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:51:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:51:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:51:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:51:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:51:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:51:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:51:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:51:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:51:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:51:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:51:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:51:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:51:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:51:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:51:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:51:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:51:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:51:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:51:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:51:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:51:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:51:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:51:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2024-09-13T09:51:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-13T09:51:58.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:51:58.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 2 2024-09-13T09:52:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:52:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:52:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-09-13T09:52:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:52:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:52:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:52:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:52:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:52:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:52:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:52:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:52:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:52:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:52:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:52:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:52:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:52:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:52:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:52:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:52:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-09-13T09:52:00.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:52:00.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:52:00.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:52:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2024-09-13T09:52:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:52:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:52:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:52:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:52:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:52:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:52:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:52:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:52:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:52:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:52:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:52:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:52:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:52:00.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:00.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:00.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:52:00.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:52:00.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:52:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:52:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:52:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:52:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2024-09-13T09:52:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:52:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:52:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:52:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2024-09-13T09:52:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:52:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:52:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:52:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:52:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:52:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:52:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:52:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:52:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:52:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:52:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:52:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:52:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:52:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:52:00.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:52:00.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:52:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:52:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:52:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:52:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2024-09-13T09:52:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:52:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:52:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:52:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2024-09-13T09:52:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:52:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:52:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:52:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:52:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:52:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:52:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:52:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:52:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:52:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:52:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:52:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:52:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:52:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:52:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:52:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:52:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:52:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:52:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:52:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2024-09-13T09:52:00.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-13T09:52:00.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:52:00.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 4 2024-09-13T09:52:01.660 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-13T09:52:01.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-13T09:52:01.660 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-13T09:52:01.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-13T09:52:01.664 INFO:tasks.rbd_mirror_thrash:waiting for 103 secs before thrashing 2024-09-13T09:52:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:52:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:52:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-09-13T09:52:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:52:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:52:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:52:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:52:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:52:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:52:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:52:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:52:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:52:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:52:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:52:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:52:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:52:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:52:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:52:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:52:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-09-13T09:52:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:52:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:52:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:52:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2024-09-13T09:52:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:52:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:52:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:52:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:52:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:52:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:52:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:52:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:52:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:52:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:52:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:52:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:52:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:52:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:52:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:52:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:52:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:52:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:52:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:52:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2024-09-13T09:52:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:52:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:52:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:52:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2024-09-13T09:52:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:52:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:52:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:52:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:52:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:52:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:52:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:52:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:52:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:52:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:52:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:52:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:52:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:52:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:52:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:52:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:52:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:52:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:52:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:52:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2024-09-13T09:52:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:52:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:52:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:52:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2024-09-13T09:52:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:52:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:52:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:52:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:52:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:52:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:52:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:52:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:52:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:52:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:52:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:52:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:52:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:52:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:52:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:52:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:52:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:52:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:52:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:52:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2024-09-13T09:52:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:52:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:52:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:52:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:52:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 8 2024-09-13T09:52:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:52:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:52:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-09-13T09:52:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:52:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:52:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:52:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:52:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:52:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:52:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:52:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:52:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:52:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:52:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:52:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:52:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:52:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:52:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:52:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:52:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-09-13T09:52:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:52:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:52:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:52:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2024-09-13T09:52:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:52:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:52:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:52:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:52:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:52:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:52:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:52:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:52:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:52:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:52:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:52:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:52:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:52:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:52:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:52:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:52:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:52:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:52:13.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:52:13.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2024-09-13T09:52:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:52:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:52:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:52:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2024-09-13T09:52:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:52:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:52:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:52:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:52:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:52:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:52:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:52:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:52:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:52:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:52:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:52:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:52:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:52:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:52:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:52:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:52:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:52:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:52:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:52:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2024-09-13T09:52:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:52:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:52:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:52:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2024-09-13T09:52:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:52:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:52:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:52:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:52:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:52:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:52:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:52:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:52:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:52:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:52:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:52:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:52:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:52:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:52:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:52:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:52:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:52:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:52:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:52:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2024-09-13T09:52:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:52:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:52:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:52:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:52:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 8 2024-09-13T09:52:18.451 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5099318588242431) > weight (0.25) 2024-09-13T09:52:18.451 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8254816721639738) > weight (0.25) 2024-09-13T09:52:18.451 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7466962764237038) > weight (0.25) 2024-09-13T09:52:18.451 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7620411557410044) > weight (0.25) 2024-09-13T09:52:18.451 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2024-09-13T09:52:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:52:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:52:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-09-13T09:52:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:52:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:52:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:52:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:52:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:52:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:52:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:52:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:52:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:52:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:52:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:52:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:52:21.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:52:21.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:21.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:21.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:52:21.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:52:21.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:52:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-09-13T09:52:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:52:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:52:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:52:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2024-09-13T09:52:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:52:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:52:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:52:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:52:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:52:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:52:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:52:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:52:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:52:21.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:52:21.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:52:21.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:52:21.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2024-09-13T09:52:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:52:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:52:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:52:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2024-09-13T09:52:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:52:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:52:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:52:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:52:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:52:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:52:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:52:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:52:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:52:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:52:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:52:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:52:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:52:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:21.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:21.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:52:21.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:52:21.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:52:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:52:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:52:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:52:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2024-09-13T09:52:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:52:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:52:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:52:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2024-09-13T09:52:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:52:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:52:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:52:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:52:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:52:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:52:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:52:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:52:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:52:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:52:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:52:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:52:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:52:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:52:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:52:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:52:21.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:52:21.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:52:21.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:52:21.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2024-09-13T09:52:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:52:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:52:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:52:21.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:52:21.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 8 2024-09-13T09:52:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:52:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:52:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-09-13T09:52:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:52:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:52:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:52:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:52:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:52:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:52:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:52:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:52:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-09-13T09:52:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:52:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:52:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:52:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2024-09-13T09:52:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:52:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:52:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:52:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:52:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:52:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:52:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:52:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:52:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:52:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:52:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:52:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:52:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:52:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:52:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:52:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:52:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:52:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:52:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:52:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2024-09-13T09:52:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:52:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:52:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:52:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2024-09-13T09:52:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:2 2024-09-13T09:52:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-13T09:52:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:2 2024-09-13T09:52:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=2 2024-09-13T09:52:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-13T09:52:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=2 2024-09-13T09:52:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:52:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-13T09:52:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:2 2024-09-13T09:52:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-13T09:52:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:2 2024-09-13T09:52:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=2 2024-09-13T09:52:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-13T09:52:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=2 2024-09-13T09:52:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=2 2024-09-13T09:52:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-13T09:52:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:52:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:52:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-13T09:52:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2024-09-13T09:52:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-13T09:52:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:52:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:52:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2024-09-13T09:52:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:3 2024-09-13T09:52:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-13T09:52:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:3 2024-09-13T09:52:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=3 2024-09-13T09:52:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-13T09:52:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=3 2024-09-13T09:52:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:52:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-13T09:52:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:3 2024-09-13T09:52:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-13T09:52:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:3 2024-09-13T09:52:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=3 2024-09-13T09:52:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-13T09:52:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=3 2024-09-13T09:52:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=3 2024-09-13T09:52:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-13T09:52:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:52:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:52:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-13T09:52:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2024-09-13T09:52:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:52:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:52:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:52:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:52:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:52:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:52:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:52:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:52:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2024-09-13T09:52:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:52:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:52:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:52:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:52:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2024-09-13T09:52:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:52:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:52:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_47.status 2024-09-13T09:52:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_47.status 2024-09-13T09:52:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:52:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_47.status 2024-09-13T09:52:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_47.status 2024-09-13T09:52:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_47.status 2024-09-13T09:52:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:52:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_47.status 2024-09-13T09:52:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2024-09-13T09:52:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:52:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2024-09-13T09:52:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 211 2024-09-13T09:52:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_47.status 2024-09-13T09:52:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:52:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_47.status 2024-09-13T09:52:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2024-09-13T09:52:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:52:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:52:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=211 2024-09-13T09:52:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:52:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:52:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:52:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_47.status 2024-09-13T09:52:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:52:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_47.status 2024-09-13T09:52:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-13T09:52:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:52:30.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:52:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=114 2024-09-13T09:52:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 211 = 114 2024-09-13T09:52:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:52:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:52:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_47.status 2024-09-13T09:52:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:52:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_47.status 2024-09-13T09:52:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-13T09:52:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:52:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:52:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:52:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:52:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:52:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:52:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:52:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 211 = '' 2024-09-13T09:52:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:52:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:52:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_47.status 2024-09-13T09:52:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:52:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_47.status 2024-09-13T09:52:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-13T09:52:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:52:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:52:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:52:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:52:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:52:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:52:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:52:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 211 = '' 2024-09-13T09:52:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:52:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:52:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_47.status 2024-09-13T09:52:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:52:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_47.status 2024-09-13T09:52:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-13T09:52:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:52:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:52:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=211 2024-09-13T09:52:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 211 = 211 2024-09-13T09:52:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:52:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-09-13T09:52:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:52:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:52:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:52:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:52:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:52:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-09-13T09:52:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_47': 2024-09-13T09:52:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:52:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:52:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:52:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12ac1a315463 2024-09-13T09:52:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12ac1a315463 2024-09-13T09:52:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:52:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:52:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:52:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:52:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:43:54 2024 2024-09-13T09:52:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:43:54 2024 2024-09-13T09:52:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:43:54 2024 2024-09-13T09:52:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:52:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:52:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:52:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 89dad582-4466-4ae2-9cac-bfea345b1268 2024-09-13T09:52:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:52:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:52:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_47 snap 2024-09-13T09:52:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:52:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:52:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:52:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_47.export 2024-09-13T09:52:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_47.export 2024-09-13T09:52:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_47.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_47.export 2024-09-13T09:52:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_47.export 2024-09-13T09:52:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:52:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_47.export 2024-09-13T09:52:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:52:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_47.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_47.export 2024-09-13T09:52:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_47.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_47.export 2024-09-13T09:52:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:52:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:52:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_48 2024-09-13T09:52:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_48 snap 2024-09-13T09:52:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:52:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:52:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:52:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-09-13T09:52:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:52:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-09-13T09:52:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:52:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:52:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:52:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:52:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:52:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-09-13T09:52:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:52:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:52:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:52:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:52:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-09-13T09:52:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:52:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:52:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:52:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:52:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:52:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:52:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:52:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:52:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:52:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-09-13T09:52:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:52:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:52:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:52:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:37.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:37.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:52:37.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:52:37.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:37.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:37.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:52:37.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:52:37.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:52:37.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:52:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:52:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:52:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:52:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:52:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:52:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:52:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:52:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-09-13T09:52:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-13T09:52:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:52:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:52:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2024-09-13T09:52:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:52:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:52:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:52:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:52:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:52:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:52:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:52:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:52:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:52:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:52:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:52:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:52:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:52:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:52:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:52:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:52:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:52:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:52:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:52:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2024-09-13T09:52:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:52:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:52:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:52:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:52:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:52:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:52:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:52:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:52:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2024-09-13T09:52:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:52:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:52:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:52:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:52:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2024-09-13T09:52:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:52:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:52:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_48.status 2024-09-13T09:52:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_48.status 2024-09-13T09:52:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:52:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_48.status 2024-09-13T09:52:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_48.status 2024-09-13T09:52:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_48.status 2024-09-13T09:52:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:52:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_48.status 2024-09-13T09:52:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2024-09-13T09:52:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:52:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2024-09-13T09:52:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 213 2024-09-13T09:52:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_48.status 2024-09-13T09:52:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:52:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_48.status 2024-09-13T09:52:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2024-09-13T09:52:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:52:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:52:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=213 2024-09-13T09:52:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:52:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:52:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:52:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_48.status 2024-09-13T09:52:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:52:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_48.status 2024-09-13T09:52:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-13T09:52:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:52:38.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:52:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=115 2024-09-13T09:52:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 213 = 115 2024-09-13T09:52:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:52:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:52:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_48.status 2024-09-13T09:52:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:52:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_48.status 2024-09-13T09:52:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-13T09:52:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:52:39.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:52:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:52:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:52:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:52:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:52:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:52:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 213 = '' 2024-09-13T09:52:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:52:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:52:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_48.status 2024-09-13T09:52:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:52:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_48.status 2024-09-13T09:52:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-13T09:52:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:52:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:52:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:52:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:52:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:52:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:52:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:52:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 213 = '' 2024-09-13T09:52:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:52:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:52:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_48.status 2024-09-13T09:52:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:52:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_48.status 2024-09-13T09:52:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-13T09:52:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:52:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:52:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=213 2024-09-13T09:52:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 213 = 213 2024-09-13T09:52:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:52:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-09-13T09:52:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:52:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:52:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:52:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:52:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:52:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-09-13T09:52:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_48': 2024-09-13T09:52:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:52:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:52:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:52:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12acd752d056 2024-09-13T09:52:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12acd752d056 2024-09-13T09:52:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:52:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:52:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:52:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:52:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:44:01 2024 2024-09-13T09:52:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:44:01 2024 2024-09-13T09:52:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:44:01 2024 2024-09-13T09:52:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:52:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:52:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:52:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 186c00be-d8ec-4acc-8296-30f83c1b7ec8 2024-09-13T09:52:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:52:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:52:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_48 snap 2024-09-13T09:52:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:52:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:52:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:52:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_48.export 2024-09-13T09:52:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_48.export 2024-09-13T09:52:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_48.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_48.export 2024-09-13T09:52:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_48.export 2024-09-13T09:52:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:52:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_48.export 2024-09-13T09:52:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:52:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_48.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_48.export 2024-09-13T09:52:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_48.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_48.export 2024-09-13T09:52:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:52:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:52:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_49 2024-09-13T09:52:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_49 snap 2024-09-13T09:52:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:52:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:52:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:52:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-09-13T09:52:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:52:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-09-13T09:52:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:52:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:52:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:52:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:52:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:52:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-09-13T09:52:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:52:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:52:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:52:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:52:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-09-13T09:52:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:52:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:52:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:52:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:52:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:52:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:52:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:52:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:52:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:52:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-09-13T09:52:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:52:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:52:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:52:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:52:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:52:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:52:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:52:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:52:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:52:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:52:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:52:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:52:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:52:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:52:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:52:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:52:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-09-13T09:52:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:52:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:52:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:52:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:52:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:52:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:52:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:52:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:52:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2024-09-13T09:52:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:52:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:52:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:52:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:52:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2024-09-13T09:52:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:52:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:52:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_49.status 2024-09-13T09:52:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_49.status 2024-09-13T09:52:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:52:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_49.status 2024-09-13T09:52:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_49.status 2024-09-13T09:52:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_49.status 2024-09-13T09:52:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:52:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_49.status 2024-09-13T09:52:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2024-09-13T09:52:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:52:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2024-09-13T09:52:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 215 2024-09-13T09:52:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_49.status 2024-09-13T09:52:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:52:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_49.status 2024-09-13T09:52:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2024-09-13T09:52:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:52:46.365 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:52:46.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=215 2024-09-13T09:52:46.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:52:46.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:52:46.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:52:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_49.status 2024-09-13T09:52:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:52:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_49.status 2024-09-13T09:52:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-13T09:52:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:52:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:52:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:52:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:52:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:52:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:52:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:52:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 215 = '' 2024-09-13T09:52:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:52:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:52:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_49.status 2024-09-13T09:52:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:52:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_49.status 2024-09-13T09:52:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-13T09:52:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:52:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:52:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:52:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:52:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:52:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:52:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:52:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 215 = '' 2024-09-13T09:52:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:52:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:52:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_49.status 2024-09-13T09:52:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:52:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_49.status 2024-09-13T09:52:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-13T09:52:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:52:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:52:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=215 2024-09-13T09:52:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 215 = 215 2024-09-13T09:52:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:52:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-09-13T09:52:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:52:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:52:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:52:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:52:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:52:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-09-13T09:52:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_49': 2024-09-13T09:52:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:52:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:52:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:52:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 10c5eed4a48f 2024-09-13T09:52:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.10c5eed4a48f 2024-09-13T09:52:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:52:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:52:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:52:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:52:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:44:11 2024 2024-09-13T09:52:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:44:11 2024 2024-09-13T09:52:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:44:11 2024 2024-09-13T09:52:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:52:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:52:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:52:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: a0524d8d-2e9d-4921-a7cd-cfc9cc8a519c 2024-09-13T09:52:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:52:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:52:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_49 snap 2024-09-13T09:52:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:52:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:52:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:52:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_49.export 2024-09-13T09:52:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_49.export 2024-09-13T09:52:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_49.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_49.export 2024-09-13T09:52:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_49.export 2024-09-13T09:52:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:52:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_49.export 2024-09-13T09:52:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:52:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_49.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_49.export 2024-09-13T09:52:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_49.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_49.export 2024-09-13T09:52:49.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:52:49.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:52:49.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_50 2024-09-13T09:52:49.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_snap cluster2 mirror image_50 snap 2024-09-13T09:52:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:52:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:52:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-09-13T09:52:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-09-13T09:52:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:52:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-09-13T09:52:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:52:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=started 2024-09-13T09:52:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:52:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:52:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:52:51.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-09-13T09:52:51.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:51.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:51.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:52:51.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=started 2024-09-13T09:52:51.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_result 2024-09-13T09:52:51.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=stopped 2024-09-13T09:52:51.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-09-13T09:52:51.598 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_instance=cluster1 2024-09-13T09:52:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -i state 2024-09-13T09:52:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:52:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance=cluster1 2024-09-13T09:52:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local loop_instance 2024-09-13T09:52:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-13T09:52:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sleep 0 2024-09-13T09:52:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-13T09:52:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ seq 0 3 2024-09-13T09:52:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-09-13T09:52:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:0 2024-09-13T09:52:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-13T09:52:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:0 2024-09-13T09:52:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:52:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-13T09:52:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=0 2024-09-13T09:52:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:52:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-13T09:52:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:0 2024-09-13T09:52:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-13T09:52:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:0 2024-09-13T09:52:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=0 2024-09-13T09:52:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-13T09:52:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=0 2024-09-13T09:52:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=0 2024-09-13T09:52:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-13T09:52:51.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:51.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:51.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-13T09:52:51.622 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-09-13T09:52:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-13T09:52:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd mirror status : get status for rbd mirror 2024-09-13T09:52:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:admin_socket: invalid command 2024-09-13T09:52:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T09:52:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-09-13T09:52:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1:1 2024-09-13T09:52:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ shift 2024-09-13T09:52:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance 2024-09-13T09:52:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-13T09:52:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local val=cluster1:1 2024-09-13T09:52:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster_var_name=cluster 2024-09-13T09:52:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local instance_var_name=instance 2024-09-13T09:52:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:52:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=1 2024-09-13T09:52:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-13T09:52:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval cluster=cluster1 2024-09-13T09:52:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ eval instance=1 2024-09-13T09:52:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:52:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-13T09:52:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local local_cluster=cluster1:1 2024-09-13T09:52:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster=cluster1 2024-09-13T09:52:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance 2024-09-13T09:52:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-13T09:52:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local val=cluster1:1 2024-09-13T09:52:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local cluster_var_name=local_cluster 2024-09-13T09:52:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ local instance_var_name=instance 2024-09-13T09:52:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ cluster=cluster1 2024-09-13T09:52:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ instance=1 2024-09-13T09:52:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-13T09:52:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval local_cluster=cluster1 2024-09-13T09:52:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ local_cluster=cluster1 2024-09-13T09:52:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ eval instance=1 2024-09-13T09:52:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ instance=1 2024-09-13T09:52:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-13T09:52:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:52:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:52:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-13T09:52:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-09-13T09:52:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ return 0 2024-09-13T09:52:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ status_result=' "state": "Replaying"' 2024-09-13T09:52:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo ' "state": "Replaying"' 2024-09-13T09:52:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ grep -i Replaying 2024-09-13T09:52:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: "state": "Replaying" 2024-09-13T09:52:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=started 2024-09-13T09:52:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test started = started 2024-09-13T09:52:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:52:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2024-09-13T09:52:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:52:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:52:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = journal ']' 2024-09-13T09:52:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' snapshot = snapshot ']' 2024-09-13T09:52:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2024-09-13T09:52:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_cluster=cluster1 2024-09-13T09:52:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:52:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster2-mirror-image_50.status 2024-09-13T09:52:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster2-mirror-image_50.status 2024-09-13T09:52:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:52:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local status_log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_50.status 2024-09-13T09:52:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ mkfname cluster1-mirror-image_50.status 2024-09-13T09:52:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ echo cluster1-mirror-image_50.status 2024-09-13T09:52:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-13T09:52:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local local_status_log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_50.status 2024-09-13T09:52:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2024-09-13T09:52:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:52:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2024-09-13T09:52:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Snapshot ID: 217 2024-09-13T09:52:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_50.status 2024-09-13T09:52:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:52:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_50.status 2024-09-13T09:52:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2024-09-13T09:52:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:52:52.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-13T09:52:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snapshot_id=217 2024-09-13T09:52:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:52:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:52:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.2 2024-09-13T09:52:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_50.status 2024-09-13T09:52:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:52:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_50.status 2024-09-13T09:52:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-13T09:52:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:52:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:52:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=117 2024-09-13T09:52:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 217 = 117 2024-09-13T09:52:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:52:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.4 2024-09-13T09:52:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_50.status 2024-09-13T09:52:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:52:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_50.status 2024-09-13T09:52:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-13T09:52:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:52:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:52:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=117 2024-09-13T09:52:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 217 = 117 2024-09-13T09:52:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:52:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.8 2024-09-13T09:52:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_50.status 2024-09-13T09:52:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:52:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_50.status 2024-09-13T09:52:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-13T09:52:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:52:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:52:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:52:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:-:1.1: Document is empty 2024-09-13T09:52:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: 2024-09-13T09:52:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:^ 2024-09-13T09:52:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id= 2024-09-13T09:52:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 217 = '' 2024-09-13T09:52:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-13T09:52:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1.6 2024-09-13T09:52:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_50.status 2024-09-13T09:52:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:52:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local log=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_50.status 2024-09-13T09:52:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-13T09:52:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-13T09:52:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-13T09:52:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local primary_snapshot_id=217 2024-09-13T09:52:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 217 = 217 2024-09-13T09:52:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:52:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-09-13T09:52:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:52:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:52:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:52:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:52:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-13T09:52:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 1 2024-09-13T09:52:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-09-13T09:52:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_50': 2024-09-13T09:52:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 128 MiB in 32 objects 2024-09-13T09:52:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:52:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 2 2024-09-13T09:52:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 12822dfaeba5 2024-09-13T09:52:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.12822dfaeba5 2024-09-13T09:52:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:52:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock, non-primary 2024-09-13T09:52:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:52:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:52:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:44:19 2024 2024-09-13T09:52:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:44:19 2024 2024-09-13T09:52:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:44:19 2024 2024-09-13T09:52:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: protected: False 2024-09-13T09:52:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:52:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:52:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 6deefad4-7356-4a90-996d-f561ce31e3b2 2024-09-13T09:52:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: false 2024-09-13T09:52:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:52:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ compare_image_snaps mirror image_50 snap 2024-09-13T09:52:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:52:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap_name=snap 2024-09-13T09:52:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local ret=0 2024-09-13T09:52:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local rmt_export=/tmp/tmp.V7YewHEkvl/cluster2-mirror-image_50.export 2024-09-13T09:52:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local loc_export=/tmp/tmp.V7YewHEkvl/cluster1-mirror-image_50.export 2024-09-13T09:52:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_50.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_50.export 2024-09-13T09:52:56.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_50.export 2024-09-13T09:52:57.083 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:52:57.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_50.export 2024-09-13T09:52:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T09:52:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cmp /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_50.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_50.export 2024-09-13T09:52:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -f /tmp/tmp.V7YewHEkvl/cluster2-mirror-image_50.export /tmp/tmp.V7YewHEkvl/cluster1-mirror-image_50.export 2024-09-13T09:52:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:52:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ testlog 'TEST: delete many images' 2024-09-13T09:52:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee -a /tmp/tmp.V7YewHEkvl/rbd-mirror.test.log 2024-09-13T09:52:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ date '+%F %T' 2024-09-13T09:52:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo 2024-09-13 09:52:57 TEST: delete many images 2024-09-13T09:52:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:2024-09-13 09:52:57 TEST: delete many images 2024-09-13T09:52:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ seq 1 50 2024-09-13T09:52:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:52:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_1 2024-09-13T09:52:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-09-13T09:52:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:52:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:52:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-09-13T09:52:57.713 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:52:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-13T09:52:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:52:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:52:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:52:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_1 2024-09-13T09:52:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:52:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-13T09:52:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-13T09:52:57.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:52:57.865+0000 7f67211c8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5589c9ab5520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:52:57.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:52:57.865+0000 7f67281d6640 -1 rbd::mirror::ImageReplayer: 0x5589c8b59400 [3/6c1c05c0-9a83-4822-98dd-9fa3a496df28] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:52:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:52:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:52:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:52:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_2 2024-09-13T09:52:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-09-13T09:52:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:52:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:52:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-09-13T09:52:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:52:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-13T09:52:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:52:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:52:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:52:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_2 2024-09-13T09:52:59.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:52:59.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:52:59.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:52:59.726 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-13T09:52:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-13T09:52:59.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:52:59.878+0000 7efcbeeec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbca0151e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:52:59.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:52:59.878+0000 7efcc56f9640 -1 rbd::mirror::ImageReplayer: 0x55dbc94e1180 [3/c4b5aac5-000e-4a00-bd35-23bce99cad5f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_3 2024-09-13T09:53:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-09-13T09:53:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:53:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-09-13T09:53:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-13T09:53:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:53:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_3 2024-09-13T09:53:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:53:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-13T09:53:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-13T09:53:01.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:53:01.884+0000 7fb8dd381640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55594db50b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:01.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:53:01.884+0000 7fb8e3b8e640 -1 rbd::mirror::ImageReplayer: 0x55594ccdbb80 [3/c99ca9ae-d4dc-48db-a446-dd6058760fec] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:02.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_4 2024-09-13T09:53:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-09-13T09:53:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:53:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-09-13T09:53:03.622 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2024-09-13T09:53:03.620+0000 7ff85b706640 -1 reset not still connected to 0x5568f5fc8ea0 2024-09-13T09:53:03.624 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2024-09-13T09:53:03.622+0000 7ff85b706640 -1 reset not still connected to 0x5568fa149110 2024-09-13T09:53:03.624 INFO:tasks.ceph.cluster2.osd.2.smithi190.stderr:2024-09-13T09:53:03.623+0000 7ff85b706640 -1 reset not still connected to 0x5569146711e0 2024-09-13T09:53:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-13T09:53:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:53:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_4 2024-09-13T09:53:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:53:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-13T09:53:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-13T09:53:03.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:53:03.885+0000 7fb8ddb82640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55594db7c340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:04.245 INFO:tasks.ceph.cluster2.osd.0.smithi190.stderr:2024-09-13T09:53:04.244+0000 7f00cac16640 -1 reset not still connected to 0x55a1b3c2c680 2024-09-13T09:53:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_5 2024-09-13T09:53:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-09-13T09:53:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:53:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-09-13T09:53:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-13T09:53:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:53:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:05.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_5 2024-09-13T09:53:05.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:05.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:05.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:53:05.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-13T09:53:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-13T09:53:05.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:53:05.337+0000 7f67211c8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5589c9ab8820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:05.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:53:05.337+0000 7f67279d5640 -1 rbd::mirror::ImageReplayer: 0x5589c8b58c80 [3/225d4f13-dab0-4960-904b-d2c54fac25f4] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_6 2024-09-13T09:53:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-09-13T09:53:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:53:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-09-13T09:53:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-13T09:53:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:53:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_6 2024-09-13T09:53:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:53:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-13T09:53:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-13T09:53:07.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:53:07.346+0000 7f67211c8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5589c9ab89c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:07.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:53:07.346+0000 7f67279d5640 -1 rbd::mirror::ImageReplayer: 0x5589c8b58a00 [3/140af9b2-8074-4c94-96e5-2095bff8e54c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_7 2024-09-13T09:53:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-09-13T09:53:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:53:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-09-13T09:53:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-13T09:53:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:53:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_7 2024-09-13T09:53:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:53:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-13T09:53:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-13T09:53:09.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:53:09.347+0000 7fb8ddb82640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55594db7c1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:09.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:53:09.347+0000 7fb8e438f640 -1 rbd::mirror::ImageReplayer: 0x55594ccda280 [3/05fdfd86-7dcf-47af-bad1-344a4f52a3bd] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_8 2024-09-13T09:53:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-09-13T09:53:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:53:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-09-13T09:53:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:11.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-13T09:53:11.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:11.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:11.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:53:11.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:11.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:11.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_8 2024-09-13T09:53:11.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:11.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:11.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:53:11.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-13T09:53:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-13T09:53:11.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:53:11.368+0000 7efcbeeec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbca0156c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:11.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:53:11.368+0000 7efcc56f9640 -1 rbd::mirror::ImageReplayer: 0x55dbc94e0780 [3/8a9c351e-2826-4d1c-9d73-5faec463921f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:11.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:53:11.575+0000 7efcbeeec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbc9e6e680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:11.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:53:11.575+0000 7efcc56f9640 -1 rbd::mirror::ImageReplayer: 0x55dbc94e0780 [3/8a9c351e-2826-4d1c-9d73-5faec463921f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_9 2024-09-13T09:53:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-09-13T09:53:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:53:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-09-13T09:53:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-13T09:53:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:53:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_9 2024-09-13T09:53:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:53:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-13T09:53:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-13T09:53:13.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:53:13.363+0000 7efcbe6eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbca014d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:13.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:53:13.363+0000 7efcc56f9640 -1 rbd::mirror::ImageReplayer: 0x55dbc94e1900 [3/ce3d15a2-54a3-4901-a2c5-f75b935cc27e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_10 2024-09-13T09:53:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-09-13T09:53:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:53:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-09-13T09:53:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:15.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-13T09:53:15.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:15.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:15.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:53:15.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:15.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:15.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_10 2024-09-13T09:53:15.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:15.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:15.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:53:15.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-13T09:53:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-13T09:53:15.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:53:15.583+0000 7f67211c8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5589c9ab8b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:15.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:53:15.583+0000 7f67279d5640 -1 rbd::mirror::ImageReplayer: 0x5589c945c280 [3/dd6c9225-99be-46ca-b40c-23bfa9e8daa3] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_11 2024-09-13T09:53:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-09-13T09:53:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:53:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-09-13T09:53:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_11 2024-09-13T09:53:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:53:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_11 2024-09-13T09:53:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:53:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-09-13T09:53:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-09-13T09:53:17.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:53:17.383+0000 7efcbeeec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbca015380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:17.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:53:17.383+0000 7efcc56f9640 -1 rbd::mirror::ImageReplayer: 0x55dbc94e1680 [3/e78a9a0a-006b-4a90-8dde-2667b994e2f6] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:18.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_12 2024-09-13T09:53:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-09-13T09:53:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:53:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-09-13T09:53:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_12 2024-09-13T09:53:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:53:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_12 2024-09-13T09:53:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:53:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-09-13T09:53:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-09-13T09:53:19.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:53:19.335+0000 7f67211c8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5589c9ab8d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:19.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:53:19.335+0000 7f67279d5640 -1 rbd::mirror::ImageReplayer: 0x5589c8b58780 [3/d677823f-dea5-4da8-94e6-42070cc16fc2] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_13 2024-09-13T09:53:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-09-13T09:53:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:53:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-09-13T09:53:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_13 2024-09-13T09:53:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:53:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:21.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_13 2024-09-13T09:53:21.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:21.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:21.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:53:21.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-09-13T09:53:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-09-13T09:53:21.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:53:21.608+0000 7fb782d43640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f264ebb6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:21.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:53:21.608+0000 7fb788d4f640 -1 rbd::mirror::ImageReplayer: 0x55f264a60280 [3/fa86eec0-62a5-41fc-9cc3-aa9334a25549] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_14 2024-09-13T09:53:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-09-13T09:53:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:53:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-09-13T09:53:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_14 2024-09-13T09:53:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:53:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_14 2024-09-13T09:53:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:53:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-09-13T09:53:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-09-13T09:53:23.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:53:23.414+0000 7fb782d43640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f264eba000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:23.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:53:23.414+0000 7fb788d4f640 -1 rbd::mirror::ImageReplayer: 0x55f264552c80 [3/393ab671-e876-42d3-878c-4b6c73393fe3] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_15 2024-09-13T09:53:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-09-13T09:53:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:53:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-09-13T09:53:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_15 2024-09-13T09:53:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:53:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_15 2024-09-13T09:53:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:53:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-09-13T09:53:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-09-13T09:53:25.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:53:25.421+0000 7fb782542640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f26442f520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:25.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:53:25.421+0000 7fb789550640 -1 rbd::mirror::ImageReplayer: 0x55f264553180 [3/22b97631-339a-40e4-93c4-03fc33081710] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_16 2024-09-13T09:53:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-09-13T09:53:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:53:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-09-13T09:53:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_16 2024-09-13T09:53:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:53:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_16 2024-09-13T09:53:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:53:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-09-13T09:53:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-09-13T09:53:27.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:53:27.436+0000 7efcbeeec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbc9e6fba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:27.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:53:27.436+0000 7efcc56f9640 -1 rbd::mirror::ImageReplayer: 0x55dbc94e1400 [3/cd59b635-c822-4dda-a6e6-0e788a2c4725] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_17 2024-09-13T09:53:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-09-13T09:53:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:53:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-09-13T09:53:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_17 2024-09-13T09:53:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:53:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_17 2024-09-13T09:53:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:53:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-09-13T09:53:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-09-13T09:53:29.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:53:29.350+0000 7f67211c8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5589c9ab8ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:29.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:53:29.350+0000 7f67279d5640 -1 rbd::mirror::ImageReplayer: 0x5589c945cc80 [3/fc6f16a3-057d-4588-8097-7af12a77aa44] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:30.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_18 2024-09-13T09:53:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-09-13T09:53:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:53:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-09-13T09:53:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_18 2024-09-13T09:53:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:53:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_18 2024-09-13T09:53:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:53:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-09-13T09:53:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-09-13T09:53:31.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:53:31.463+0000 7efcbe6eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbc959ad00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:31.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:53:31.463+0000 7efcc56f9640 -1 rbd::mirror::ImageReplayer: 0x55dbc94e1b80 [3/d7e04b0a-b54c-4532-b814-c9ae04053bfa] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:32.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_19 2024-09-13T09:53:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-09-13T09:53:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:53:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-09-13T09:53:33.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_19 2024-09-13T09:53:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:53:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_19 2024-09-13T09:53:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:53:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-09-13T09:53:33.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-09-13T09:53:33.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:53:33.444+0000 7fb782d43640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f26442f6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:33.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:53:33.444+0000 7fb789550640 -1 rbd::mirror::ImageReplayer: 0x55f264a60000 [3/98a2a84e-2461-434e-b3e3-a5de2949eb05] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_20 2024-09-13T09:53:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-09-13T09:53:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:53:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-09-13T09:53:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_20 2024-09-13T09:53:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:53:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_20 2024-09-13T09:53:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:53:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-09-13T09:53:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-09-13T09:53:35.452 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5834782559550132) > weight (0.25) 2024-09-13T09:53:35.453 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5132197231522138) > weight (0.25) 2024-09-13T09:53:35.453 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.34840617981379607) > weight (0.25) 2024-09-13T09:53:35.453 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5337112394658364) > weight (0.25) 2024-09-13T09:53:35.453 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-09-13T09:53:35.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:53:35.456+0000 7fb8dd381640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55594d51c680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:35.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:53:35.456+0000 7fb8e438f640 -1 rbd::mirror::ImageReplayer: 0x55594ccdb680 [3/ab0efa63-cc31-4345-9caf-c5a58d606330] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_21 2024-09-13T09:53:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-09-13T09:53:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:53:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-09-13T09:53:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_21 2024-09-13T09:53:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:53:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_21 2024-09-13T09:53:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:53:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-09-13T09:53:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-09-13T09:53:37.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:53:37.467+0000 7f67219c9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5589c9b1b1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:37.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:53:37.467+0000 7f67281d6640 -1 rbd::mirror::ImageReplayer: 0x5589c8b58280 [3/47058edf-e75b-483b-aaa7-812779184a08] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_22 2024-09-13T09:53:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-09-13T09:53:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:53:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-09-13T09:53:39.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_22 2024-09-13T09:53:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:53:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_22 2024-09-13T09:53:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:53:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-09-13T09:53:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-09-13T09:53:39.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:53:39.338+0000 7fb8ddb82640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55594db50ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:39.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:53:39.338+0000 7fb8e3b8e640 -1 rbd::mirror::ImageReplayer: 0x55594ccc8280 [3/9cf71ada-0f10-4f47-91dd-9b7f8293279d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_23 2024-09-13T09:53:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-09-13T09:53:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:53:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-09-13T09:53:41.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_23 2024-09-13T09:53:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:53:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_23 2024-09-13T09:53:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:53:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-09-13T09:53:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-09-13T09:53:41.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:53:41.484+0000 7fb782d43640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f26442f860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:41.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:53:41.484+0000 7fb789550640 -1 rbd::mirror::ImageReplayer: 0x55f264553900 [3/b1f94b49-8135-472a-8920-462d2d90e565] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_24 2024-09-13T09:53:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-09-13T09:53:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:53:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-09-13T09:53:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_24 2024-09-13T09:53:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:53:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_24 2024-09-13T09:53:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:53:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-09-13T09:53:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-09-13T09:53:43.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:53:43.493+0000 7efcbe6eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbca054000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:43.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:53:43.493+0000 7efcc4ef8640 -1 rbd::mirror::ImageReplayer: 0x55dbc94e0c80 [3/aa7749ba-cd93-4a24-906c-a357a64f71e8] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_25 2024-09-13T09:53:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-09-13T09:53:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:53:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-09-13T09:53:44.665 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.256572552140766) > weight (0.25) 2024-09-13T09:53:44.665 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6661813528532272) > weight (0.25) 2024-09-13T09:53:44.665 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7054719535969242) > weight (0.25) 2024-09-13T09:53:44.665 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.48422192073715353) > weight (0.25) 2024-09-13T09:53:44.665 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2024-09-13T09:53:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_25 2024-09-13T09:53:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:53:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_25 2024-09-13T09:53:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:53:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-09-13T09:53:45.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-09-13T09:53:45.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:53:45.500+0000 7f67219c9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5589c9b1b380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:45.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:53:45.500+0000 7f67281d6640 -1 rbd::mirror::ImageReplayer: 0x5589c8b58500 [3/b4563f80-e68a-4c65-bea2-f31514c46d7c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_26 2024-09-13T09:53:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-09-13T09:53:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:53:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-09-13T09:53:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_26 2024-09-13T09:53:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:53:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_26 2024-09-13T09:53:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:53:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-09-13T09:53:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-09-13T09:53:47.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:53:47.518+0000 7fb8ddb82640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55594d51d380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:47.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:53:47.518+0000 7fb8e438f640 -1 rbd::mirror::ImageReplayer: 0x55594ccdaf00 [3/9c358941-453b-4f18-9eae-0b0fdf040056] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_27 2024-09-13T09:53:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-09-13T09:53:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:53:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-09-13T09:53:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_27 2024-09-13T09:53:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:53:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_27 2024-09-13T09:53:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:53:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-09-13T09:53:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-09-13T09:53:49.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:53:49.349+0000 7fb782542640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f2647229c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:49.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:53:49.349+0000 7fb788d4f640 -1 rbd::mirror::ImageReplayer: 0x55f264553680 [3/5ad6932e-f05d-4de5-89fb-186e6b5e26b1] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_28 2024-09-13T09:53:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-09-13T09:53:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:53:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-09-13T09:53:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_28 2024-09-13T09:53:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:53:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_28 2024-09-13T09:53:51.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:51.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:51.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:53:51.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-09-13T09:53:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-09-13T09:53:51.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:53:51.527+0000 7fb8dd381640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55594db51040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:51.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:53:51.527+0000 7fb8e3b8e640 -1 rbd::mirror::ImageReplayer: 0x55594ccc8000 [3/f0cfbdbd-fc4d-4303-a84d-1580b7778f89] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_29 2024-09-13T09:53:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-09-13T09:53:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:53:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-09-13T09:53:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_29 2024-09-13T09:53:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:53:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_29 2024-09-13T09:53:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:53:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-09-13T09:53:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-09-13T09:53:53.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:53:53.528+0000 7fb782542640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f264f83860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_30 2024-09-13T09:53:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-09-13T09:53:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:53:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-09-13T09:53:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_30 2024-09-13T09:53:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:53:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_30 2024-09-13T09:53:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:53:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-09-13T09:53:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-09-13T09:53:55.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:53:55.539+0000 7fb782542640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f264722820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:55.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:53:55.539+0000 7fb788d4f640 -1 rbd::mirror::ImageReplayer: 0x55f264552500 [3/1b942165-b9f5-4c05-8666-d8d9900c56b6] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_31 2024-09-13T09:53:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-09-13T09:53:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:53:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-09-13T09:53:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_31 2024-09-13T09:53:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:53:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_31 2024-09-13T09:53:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:53:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-09-13T09:53:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-09-13T09:53:57.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:53:57.544+0000 7fb782542640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f264722680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:57.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:53:57.544+0000 7fb788d4f640 -1 rbd::mirror::ImageReplayer: 0x55f264552f00 [3/1bb5e3e6-31d3-4cca-8cd7-40225052afae] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:53:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:53:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:53:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:53:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_32 2024-09-13T09:53:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-09-13T09:53:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:53:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:53:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-09-13T09:53:59.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:53:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_32 2024-09-13T09:53:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:53:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:53:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:53:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_32 2024-09-13T09:53:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:53:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:53:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:53:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-09-13T09:53:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-09-13T09:53:59.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:53:59.342+0000 7efcbeeec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbca054680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:53:59.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:53:59.342+0000 7efcc4ef8640 -1 rbd::mirror::ImageReplayer: 0x55dbc94e0f00 [3/ae63b672-de66-4994-b9e2-9cc620b8e9ef] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:54:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:54:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_33 2024-09-13T09:54:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-09-13T09:54:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:54:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:54:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-09-13T09:54:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:54:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_33 2024-09-13T09:54:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:54:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:54:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:54:01.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_33 2024-09-13T09:54:01.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:01.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:01.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:54:01.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-09-13T09:54:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-09-13T09:54:01.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:54:01.358+0000 7fb782d43640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f264f83ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:54:01.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:54:01.358+0000 7fb789550640 -1 rbd::mirror::ImageReplayer: 0x55f264552780 [3/7219b237-53fc-4501-baae-6c41f736571a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:54:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:54:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_34 2024-09-13T09:54:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-09-13T09:54:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:54:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:54:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-09-13T09:54:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:54:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_34 2024-09-13T09:54:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:54:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:54:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:54:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_34 2024-09-13T09:54:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:03.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:03.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:54:03.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-09-13T09:54:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-09-13T09:54:03.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:54:03.548+0000 7fb782542640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f264bfd1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:54:03.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:54:03.548+0000 7fb788d4f640 -1 rbd::mirror::ImageReplayer: 0x55f264552a00 [3/7902759b-4c55-479e-b92d-2527b62dd9df] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:54:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:54:04.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_35 2024-09-13T09:54:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-09-13T09:54:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:54:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:54:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-09-13T09:54:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:54:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_35 2024-09-13T09:54:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:54:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:54:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:54:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_35 2024-09-13T09:54:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:54:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-09-13T09:54:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-09-13T09:54:05.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:54:05.359+0000 7f67211c8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5589c9ae24e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:54:05.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:54:05.359+0000 7f67279d5640 -1 rbd::mirror::ImageReplayer: 0x5589c8b59680 [3/6cbe51d3-590c-415c-bec2-78c533fae3f5] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:54:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:54:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_36 2024-09-13T09:54:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-09-13T09:54:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:54:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:54:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-09-13T09:54:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:54:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_36 2024-09-13T09:54:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:54:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:54:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:54:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_36 2024-09-13T09:54:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:54:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-09-13T09:54:07.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-09-13T09:54:07.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:54:07.375+0000 7f67211c8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5589c9ae2680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:54:07.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:54:07.375+0000 7f67279d5640 -1 rbd::mirror::ImageReplayer: 0x5589c945c000 [3/9f803efc-9109-44ce-9b95-114b6dc86252] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:54:08.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:54:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_37 2024-09-13T09:54:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-09-13T09:54:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:54:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:54:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-09-13T09:54:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:54:09.193 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_37 2024-09-13T09:54:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:54:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:54:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:54:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_37 2024-09-13T09:54:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:54:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-09-13T09:54:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-09-13T09:54:09.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:54:09.356+0000 7f67211c8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5589c9ae2820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:54:09.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:54:09.356+0000 7f67279d5640 -1 rbd::mirror::ImageReplayer: 0x5589c8b58f00 [3/2428288c-879f-45c0-a7c5-4f2760c5f863] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:54:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:54:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_38 2024-09-13T09:54:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-09-13T09:54:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:54:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:54:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-09-13T09:54:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:54:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_38 2024-09-13T09:54:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:54:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:54:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:54:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_38 2024-09-13T09:54:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:54:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-09-13T09:54:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-09-13T09:54:11.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:54:11.380+0000 7fb782542640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f264936b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:54:11.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:54:11.380+0000 7fb788d4f640 -1 rbd::mirror::ImageReplayer: 0x55f264553400 [3/53855d34-b4b5-4803-8724-2964e4fd7b9d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:54:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:54:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_39 2024-09-13T09:54:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-09-13T09:54:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:54:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:54:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-09-13T09:54:13.228 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:54:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_39 2024-09-13T09:54:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:54:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:54:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:54:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_39 2024-09-13T09:54:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:54:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-09-13T09:54:13.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-09-13T09:54:13.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:54:13.391+0000 7efcbeeec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbca0151e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:54:13.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:54:13.391+0000 7efcc4ef8640 -1 rbd::mirror::ImageReplayer: 0x55dbc9f1c000 [3/f2864223-3140-4220-a2fa-798f1e72b93d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:54:14.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:54:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_40 2024-09-13T09:54:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-09-13T09:54:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:54:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:54:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-09-13T09:54:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:54:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_40 2024-09-13T09:54:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:54:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:54:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:54:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_40 2024-09-13T09:54:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:54:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-09-13T09:54:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-09-13T09:54:15.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:54:15.407+0000 7f67211c8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5589c9ae29c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:54:15.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:54:15.407+0000 7f67279d5640 -1 rbd::mirror::ImageReplayer: 0x5589c8b59180 [3/80af752a-a561-44aa-a889-99b950f68127] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:54:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:54:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_41 2024-09-13T09:54:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-09-13T09:54:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:54:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:54:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-09-13T09:54:17.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:54:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_41 2024-09-13T09:54:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:54:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:54:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:54:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_41 2024-09-13T09:54:17.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:17.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:17.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:54:17.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-09-13T09:54:17.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-09-13T09:54:17.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:54:17.416+0000 7fb8dd381640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55594db7c9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:54:17.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:54:17.416+0000 7fb8e3b8e640 -1 rbd::mirror::ImageReplayer: 0x55594ccc9180 [3/80c5a437-de46-4aee-a1dd-7b12f06d8126] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:54:18.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:54:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_42 2024-09-13T09:54:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-09-13T09:54:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:54:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:54:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-09-13T09:54:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:54:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_42 2024-09-13T09:54:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:54:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:54:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:54:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_42 2024-09-13T09:54:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:54:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-09-13T09:54:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-09-13T09:54:19.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:54:19.359+0000 7efcbe6eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbc96c3520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:54:19.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:54:19.359+0000 7efcc4ef8640 -1 rbd::mirror::ImageReplayer: 0x55dbc94e0280 [3/66db570e-e124-4bcd-9c58-ccbd57311102] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:54:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:54:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_43 2024-09-13T09:54:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-09-13T09:54:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:54:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:54:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-09-13T09:54:21.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:54:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_43 2024-09-13T09:54:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:54:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:54:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:54:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_43 2024-09-13T09:54:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:54:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-09-13T09:54:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-09-13T09:54:21.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:54:21.425+0000 7efcbeeec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbc959aea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:54:21.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:54:21.425+0000 7efcc4ef8640 -1 rbd::mirror::ImageReplayer: 0x55dbc94e0500 [3/69d778b5-940c-4aa1-8021-ea2d132d002e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:54:22.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:54:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_44 2024-09-13T09:54:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-09-13T09:54:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:22.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:22.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:54:22.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:54:22.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-09-13T09:54:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:54:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_44 2024-09-13T09:54:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:54:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:54:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:54:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_44 2024-09-13T09:54:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:54:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-09-13T09:54:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-09-13T09:54:23.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:54:23.434+0000 7efcbeeec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbc9e6e680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:54:23.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:54:23.434+0000 7efcc4ef8640 -1 rbd::mirror::ImageReplayer: 0x55dbc94e0a00 [3/90e5e796-ce61-474c-b054-7f1a077502a8] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:54:24.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:54:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_45 2024-09-13T09:54:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-09-13T09:54:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:54:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:54:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-09-13T09:54:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:54:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_45 2024-09-13T09:54:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:54:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:54:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:54:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_45 2024-09-13T09:54:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:54:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-09-13T09:54:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-09-13T09:54:25.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:54:25.437+0000 7fb8dd381640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55594d787d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:54:25.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:54:25.437+0000 7fb8e3b8e640 -1 rbd::mirror::ImageReplayer: 0x55594ccdac80 [3/24eb5df8-5de6-4058-952e-856c1cb902e7] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:54:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:54:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_46 2024-09-13T09:54:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-09-13T09:54:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:54:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:54:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-09-13T09:54:27.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:54:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_46 2024-09-13T09:54:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:54:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:54:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:54:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_46 2024-09-13T09:54:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:54:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-09-13T09:54:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-09-13T09:54:27.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:54:27.453+0000 7fb8dd381640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55594db089c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:54:27.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:54:27.453+0000 7fb8e3b8e640 -1 rbd::mirror::ImageReplayer: 0x55594ccdb900 [3/b2303ec4-1348-4438-91f9-318b0bbb03c1] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:54:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:54:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_47 2024-09-13T09:54:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-09-13T09:54:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:54:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:54:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-09-13T09:54:28.651 INFO:tasks.ceph.cluster1.osd.1.smithi122.stderr:2024-09-13T09:54:28.648+0000 7f8856808640 -1 reset not still connected to 0x55ae1ef1dee0 2024-09-13T09:54:28.651 INFO:tasks.ceph.cluster1.osd.1.smithi122.stderr:2024-09-13T09:54:28.648+0000 7f8856808640 -1 reset not still connected to 0x55ae20bc12b0 2024-09-13T09:54:28.652 INFO:tasks.ceph.cluster1.osd.1.smithi122.stderr:2024-09-13T09:54:28.648+0000 7f8856808640 -1 reset not still connected to 0x55ae20f9a410 2024-09-13T09:54:28.652 INFO:tasks.ceph.cluster1.osd.1.smithi122.stderr:2024-09-13T09:54:28.648+0000 7f8856808640 -1 reset not still connected to 0x55ae229141a0 2024-09-13T09:54:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:54:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_47 2024-09-13T09:54:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:54:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:54:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:54:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_47 2024-09-13T09:54:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:54:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-09-13T09:54:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-09-13T09:54:29.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:54:29.350+0000 7f67211c8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5589c9ae2b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:54:29.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:54:29.350+0000 7f67279d5640 -1 rbd::mirror::ImageReplayer: 0x5589c8b59900 [3/89dad582-4466-4ae2-9cac-bfea345b1268] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:54:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:54:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_48 2024-09-13T09:54:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-09-13T09:54:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:54:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:54:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-09-13T09:54:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:54:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_48 2024-09-13T09:54:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:54:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:54:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:54:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_48 2024-09-13T09:54:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:54:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-09-13T09:54:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-09-13T09:54:31.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:54:31.469+0000 7fb8dd381640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55594d51c340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:54:31.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:54:31.469+0000 7fb8e438f640 -1 rbd::mirror::ImageReplayer: 0x55594ccda500 [3/186c00be-d8ec-4acc-8296-30f83c1b7ec8] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:54:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:54:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_49 2024-09-13T09:54:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-09-13T09:54:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:54:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:54:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-09-13T09:54:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:54:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_49 2024-09-13T09:54:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:54:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:54:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:54:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_49 2024-09-13T09:54:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:54:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-09-13T09:54:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-09-13T09:54:33.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:54:33.478+0000 7fb8ddb82640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55594d51c680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:54:33.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:54:33.478+0000 7fb8e438f640 -1 rbd::mirror::ImageReplayer: 0x55594ccdb400 [3/a0524d8d-2e9d-4921-a7cd-cfc9cc8a519c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:54:33.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:54:33.568+0000 7fb8ddb82640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55594d51c9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:54:33.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:54:33.568+0000 7fb8e3b8e640 -1 rbd::mirror::ImageReplayer: 0x55594ccdb400 [3/a0524d8d-2e9d-4921-a7cd-cfc9cc8a519c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:54:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:54:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_50 2024-09-13T09:54:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-09-13T09:54:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:54:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local snap=snap 2024-09-13T09:54:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-09-13T09:54:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing snap: 100% complete...done. 2024-09-13T09:54:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_50 2024-09-13T09:54:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:54:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:54:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:54:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_50 2024-09-13T09:54:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:54:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-09-13T09:54:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-09-13T09:54:35.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:54:35.480+0000 7fb782d43640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f264240d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T09:54:35.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T09:54:35.480+0000 7fb789550640 -1 rbd::mirror::ImageReplayer: 0x55f264553b80 [3/6deefad4-7356-4a90-996d-f561ce31e3b2] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T09:54:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T09:54:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ testlog 'TEST: image deletions should propagate' 2024-09-13T09:54:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee -a /tmp/tmp.V7YewHEkvl/rbd-mirror.test.log 2024-09-13T09:54:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ date '+%F %T' 2024-09-13T09:54:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo 2024-09-13 09:54:36 TEST: image deletions should propagate 2024-09-13T09:54:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:2024-09-13 09:54:36 TEST: image deletions should propagate 2024-09-13T09:54:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-09-13T09:54:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_count=0 2024-09-13T09:54:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local count 2024-09-13T09:54:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local last_count=0 2024-09-13T09:54:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ true 2024-09-13T09:54:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ seq 1 40 2024-09-13T09:54:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in `seq 1 40` 2024-09-13T09:54:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 1 -ne 1 2024-09-13T09:54:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-13T09:54:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep 'images: ' 2024-09-13T09:54:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cut '-d ' -f 2 2024-09-13T09:54:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2024-09-13T09:54:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ count=0 2024-09-13T09:54:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 = 0 2024-09-13T09:54:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-09-13T09:54:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state 2024-09-13T09:54:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ seq 1 40 2024-09-13T09:54:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in `seq 1 40` 2024-09-13T09:54:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 1 -ne 1 2024-09-13T09:54:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-13T09:54:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep 'image health:' 2024-09-13T09:54:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cut '-d ' -f 3 2024-09-13T09:54:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: failed to query services: (13) Permission denied 2024-09-13T09:54:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ state=OK 2024-09-13T09:54:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test OK = ERROR 2024-09-13T09:54:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test OK = OK 2024-09-13T09:54:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ seq 1 50 2024-09-13T09:54:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_1 2024-09-13T09:54:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-13T09:54:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:54:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_1 2024-09-13T09:54:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_1 2024-09-13T09:54:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-13T09:54:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-13T09:54:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-13T09:54:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:54:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_1 2024-09-13T09:54:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_1 2024-09-13T09:54:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-13T09:54:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-13T09:54:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_2 2024-09-13T09:54:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-13T09:54:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:54:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_2 2024-09-13T09:54:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_2 2024-09-13T09:54:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-13T09:54:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-13T09:54:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-13T09:54:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:54:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_2 2024-09-13T09:54:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_2 2024-09-13T09:54:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-13T09:54:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-13T09:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_3 2024-09-13T09:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-13T09:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:54:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_3 2024-09-13T09:54:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_3 2024-09-13T09:54:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-13T09:54:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-13T09:54:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-13T09:54:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:54:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_3 2024-09-13T09:54:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_3 2024-09-13T09:54:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-13T09:54:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-13T09:54:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_4 2024-09-13T09:54:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-13T09:54:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:54:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_4 2024-09-13T09:54:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_4 2024-09-13T09:54:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-13T09:54:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-13T09:54:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-13T09:54:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:54:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_4 2024-09-13T09:54:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_4 2024-09-13T09:54:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-13T09:54:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-13T09:54:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_5 2024-09-13T09:54:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-13T09:54:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:54:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_5 2024-09-13T09:54:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_5 2024-09-13T09:54:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-13T09:54:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-13T09:54:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-13T09:54:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:54:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_5 2024-09-13T09:54:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_5 2024-09-13T09:54:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-13T09:54:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:37.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-13T09:54:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_6 2024-09-13T09:54:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-13T09:54:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:54:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_6 2024-09-13T09:54:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_6 2024-09-13T09:54:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-13T09:54:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-13T09:54:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-13T09:54:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:54:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_6 2024-09-13T09:54:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_6 2024-09-13T09:54:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-13T09:54:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-13T09:54:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_7 2024-09-13T09:54:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-13T09:54:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:54:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_7 2024-09-13T09:54:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_7 2024-09-13T09:54:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-13T09:54:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-13T09:54:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-13T09:54:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:54:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_7 2024-09-13T09:54:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_7 2024-09-13T09:54:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-13T09:54:37.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-13T09:54:37.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:37.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:37.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:37.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:37.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:37.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_8 2024-09-13T09:54:37.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-13T09:54:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:54:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_8 2024-09-13T09:54:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_8 2024-09-13T09:54:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-13T09:54:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-13T09:54:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-13T09:54:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:54:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_8 2024-09-13T09:54:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_8 2024-09-13T09:54:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-13T09:54:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-13T09:54:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_9 2024-09-13T09:54:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-13T09:54:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:54:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_9 2024-09-13T09:54:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_9 2024-09-13T09:54:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-13T09:54:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-13T09:54:37.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-13T09:54:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T09:54:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_9 2024-09-13T09:54:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_9 2024-09-13T09:54:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-13T09:54:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-13T09:54:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_10 2024-09-13T09:54:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-13T09:54:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:54:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_10 2024-09-13T09:54:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_10 2024-09-13T09:54:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-13T09:54:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-13T09:54:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-13T09:54:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T09:54:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_10 2024-09-13T09:54:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_10 2024-09-13T09:54:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-13T09:54:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-13T09:54:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_11 2024-09-13T09:54:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-09-13T09:54:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:54:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_11 2024-09-13T09:54:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_11 2024-09-13T09:54:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-13T09:54:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:38.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-13T09:54:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-09-13T09:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_11 2024-09-13T09:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_11 2024-09-13T09:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_11 2024-09-13T09:54:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-13T09:54:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-13T09:54:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_12 2024-09-13T09:54:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-09-13T09:54:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:54:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_12 2024-09-13T09:54:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_12 2024-09-13T09:54:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-13T09:54:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-13T09:54:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-09-13T09:54:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_12 2024-09-13T09:54:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_12 2024-09-13T09:54:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_12 2024-09-13T09:54:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-13T09:54:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-13T09:54:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_13 2024-09-13T09:54:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-09-13T09:54:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:54:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_13 2024-09-13T09:54:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_13 2024-09-13T09:54:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-13T09:54:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:38.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-13T09:54:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-09-13T09:54:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_13 2024-09-13T09:54:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_13 2024-09-13T09:54:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_13 2024-09-13T09:54:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-13T09:54:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-13T09:54:38.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:38.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:38.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:38.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:38.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:38.797 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_14 2024-09-13T09:54:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-09-13T09:54:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:54:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_14 2024-09-13T09:54:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_14 2024-09-13T09:54:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-13T09:54:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-13T09:54:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-09-13T09:54:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_14 2024-09-13T09:54:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_14 2024-09-13T09:54:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_14 2024-09-13T09:54:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-13T09:54:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-13T09:54:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_15 2024-09-13T09:54:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-09-13T09:54:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:54:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_15 2024-09-13T09:54:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_15 2024-09-13T09:54:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-13T09:54:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-13T09:54:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-09-13T09:54:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_15 2024-09-13T09:54:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_15 2024-09-13T09:54:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_15 2024-09-13T09:54:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-13T09:54:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-13T09:54:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_16 2024-09-13T09:54:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-09-13T09:54:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:54:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_16 2024-09-13T09:54:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_16 2024-09-13T09:54:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-13T09:54:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-13T09:54:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-09-13T09:54:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_16 2024-09-13T09:54:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_16 2024-09-13T09:54:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_16 2024-09-13T09:54:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-13T09:54:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:39.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-13T09:54:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_17 2024-09-13T09:54:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-09-13T09:54:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:54:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_17 2024-09-13T09:54:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_17 2024-09-13T09:54:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-13T09:54:39.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-13T09:54:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-09-13T09:54:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_17 2024-09-13T09:54:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_17 2024-09-13T09:54:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_17 2024-09-13T09:54:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-13T09:54:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-13T09:54:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_18 2024-09-13T09:54:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-09-13T09:54:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:54:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_18 2024-09-13T09:54:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_18 2024-09-13T09:54:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-13T09:54:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:39.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-13T09:54:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-09-13T09:54:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_18 2024-09-13T09:54:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_18 2024-09-13T09:54:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_18 2024-09-13T09:54:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-13T09:54:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-13T09:54:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_19 2024-09-13T09:54:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-09-13T09:54:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:54:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_19 2024-09-13T09:54:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_19 2024-09-13T09:54:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-13T09:54:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-13T09:54:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-09-13T09:54:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_19 2024-09-13T09:54:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_19 2024-09-13T09:54:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_19 2024-09-13T09:54:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-13T09:54:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-13T09:54:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_20 2024-09-13T09:54:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-09-13T09:54:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:54:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_20 2024-09-13T09:54:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_20 2024-09-13T09:54:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-13T09:54:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:39.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-13T09:54:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-09-13T09:54:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_20 2024-09-13T09:54:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_20 2024-09-13T09:54:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_20 2024-09-13T09:54:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-13T09:54:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:40.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-13T09:54:40.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:40.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:40.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_21 2024-09-13T09:54:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-09-13T09:54:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:54:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_21 2024-09-13T09:54:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_21 2024-09-13T09:54:40.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-13T09:54:40.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-13T09:54:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-09-13T09:54:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_21 2024-09-13T09:54:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_21 2024-09-13T09:54:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_21 2024-09-13T09:54:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-13T09:54:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:40.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-13T09:54:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_22 2024-09-13T09:54:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-09-13T09:54:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:54:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_22 2024-09-13T09:54:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_22 2024-09-13T09:54:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-13T09:54:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-13T09:54:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-09-13T09:54:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_22 2024-09-13T09:54:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_22 2024-09-13T09:54:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_22 2024-09-13T09:54:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-13T09:54:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-13T09:54:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_23 2024-09-13T09:54:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-09-13T09:54:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:54:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_23 2024-09-13T09:54:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_23 2024-09-13T09:54:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-13T09:54:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-13T09:54:40.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:40.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:40.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-09-13T09:54:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_23 2024-09-13T09:54:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_23 2024-09-13T09:54:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_23 2024-09-13T09:54:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-13T09:54:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-13T09:54:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_24 2024-09-13T09:54:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-09-13T09:54:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:54:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_24 2024-09-13T09:54:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_24 2024-09-13T09:54:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-13T09:54:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:40.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-13T09:54:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-09-13T09:54:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_24 2024-09-13T09:54:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_24 2024-09-13T09:54:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_24 2024-09-13T09:54:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-13T09:54:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-13T09:54:40.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:40.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:40.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:40.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:40.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:40.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_25 2024-09-13T09:54:40.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-09-13T09:54:40.705 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:54:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_25 2024-09-13T09:54:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_25 2024-09-13T09:54:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-13T09:54:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-13T09:54:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-09-13T09:54:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_25 2024-09-13T09:54:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_25 2024-09-13T09:54:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_25 2024-09-13T09:54:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-13T09:54:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-13T09:54:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_26 2024-09-13T09:54:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-09-13T09:54:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:54:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_26 2024-09-13T09:54:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_26 2024-09-13T09:54:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-13T09:54:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-13T09:54:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-09-13T09:54:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_26 2024-09-13T09:54:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_26 2024-09-13T09:54:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_26 2024-09-13T09:54:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-13T09:54:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-13T09:54:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_27 2024-09-13T09:54:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-09-13T09:54:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:54:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_27 2024-09-13T09:54:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_27 2024-09-13T09:54:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-13T09:54:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-13T09:54:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:41.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-09-13T09:54:41.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_27 2024-09-13T09:54:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_27 2024-09-13T09:54:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_27 2024-09-13T09:54:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-13T09:54:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-13T09:54:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_28 2024-09-13T09:54:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-09-13T09:54:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:54:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_28 2024-09-13T09:54:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_28 2024-09-13T09:54:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-13T09:54:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-13T09:54:41.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:41.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:41.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-09-13T09:54:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_28 2024-09-13T09:54:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_28 2024-09-13T09:54:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_28 2024-09-13T09:54:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-13T09:54:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-13T09:54:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_29 2024-09-13T09:54:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-09-13T09:54:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:54:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_29 2024-09-13T09:54:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_29 2024-09-13T09:54:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-13T09:54:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-13T09:54:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-09-13T09:54:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_29 2024-09-13T09:54:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_29 2024-09-13T09:54:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_29 2024-09-13T09:54:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-13T09:54:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:41.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-13T09:54:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_30 2024-09-13T09:54:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-09-13T09:54:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:54:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_30 2024-09-13T09:54:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_30 2024-09-13T09:54:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-13T09:54:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-13T09:54:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-09-13T09:54:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_30 2024-09-13T09:54:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_30 2024-09-13T09:54:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_30 2024-09-13T09:54:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-13T09:54:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-13T09:54:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_31 2024-09-13T09:54:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-09-13T09:54:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:54:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_31 2024-09-13T09:54:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_31 2024-09-13T09:54:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-13T09:54:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-13T09:54:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-09-13T09:54:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_31 2024-09-13T09:54:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_31 2024-09-13T09:54:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_31 2024-09-13T09:54:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-13T09:54:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:41.912 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-13T09:54:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_32 2024-09-13T09:54:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-09-13T09:54:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:54:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_32 2024-09-13T09:54:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_32 2024-09-13T09:54:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-13T09:54:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-13T09:54:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-09-13T09:54:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_32 2024-09-13T09:54:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_32 2024-09-13T09:54:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_32 2024-09-13T09:54:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-13T09:54:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-13T09:54:42.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:42.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:42.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:42.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:42.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:42.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_33 2024-09-13T09:54:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-09-13T09:54:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:54:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_33 2024-09-13T09:54:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_33 2024-09-13T09:54:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-13T09:54:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-13T09:54:42.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:42.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:42.124 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-09-13T09:54:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_33 2024-09-13T09:54:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_33 2024-09-13T09:54:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_33 2024-09-13T09:54:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-13T09:54:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-13T09:54:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_34 2024-09-13T09:54:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-09-13T09:54:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:54:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_34 2024-09-13T09:54:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_34 2024-09-13T09:54:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-13T09:54:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:42.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-13T09:54:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-09-13T09:54:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_34 2024-09-13T09:54:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_34 2024-09-13T09:54:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_34 2024-09-13T09:54:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-13T09:54:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-13T09:54:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_35 2024-09-13T09:54:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-09-13T09:54:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:54:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_35 2024-09-13T09:54:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_35 2024-09-13T09:54:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-13T09:54:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-13T09:54:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-09-13T09:54:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_35 2024-09-13T09:54:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_35 2024-09-13T09:54:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_35 2024-09-13T09:54:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-13T09:54:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-13T09:54:42.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_36 2024-09-13T09:54:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-09-13T09:54:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:54:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_36 2024-09-13T09:54:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_36 2024-09-13T09:54:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-13T09:54:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-13T09:54:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-09-13T09:54:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_36 2024-09-13T09:54:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_36 2024-09-13T09:54:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_36 2024-09-13T09:54:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-13T09:54:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-13T09:54:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_37 2024-09-13T09:54:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-09-13T09:54:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:54:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_37 2024-09-13T09:54:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_37 2024-09-13T09:54:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-13T09:54:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-13T09:54:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-09-13T09:54:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_37 2024-09-13T09:54:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_37 2024-09-13T09:54:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_37 2024-09-13T09:54:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-13T09:54:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-13T09:54:42.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:42.959 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_38 2024-09-13T09:54:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-09-13T09:54:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:54:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_38 2024-09-13T09:54:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:42.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_38 2024-09-13T09:54:42.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-13T09:54:42.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-13T09:54:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:43.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-09-13T09:54:43.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:43.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:43.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_38 2024-09-13T09:54:43.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:43.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:43.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:43.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:43.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_38 2024-09-13T09:54:43.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_38 2024-09-13T09:54:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-13T09:54:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-13T09:54:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_39 2024-09-13T09:54:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-09-13T09:54:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:54:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_39 2024-09-13T09:54:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_39 2024-09-13T09:54:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-13T09:54:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-13T09:54:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-09-13T09:54:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_39 2024-09-13T09:54:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_39 2024-09-13T09:54:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_39 2024-09-13T09:54:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-13T09:54:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-13T09:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_40 2024-09-13T09:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-09-13T09:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:54:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_40 2024-09-13T09:54:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_40 2024-09-13T09:54:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-13T09:54:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-13T09:54:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-09-13T09:54:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_40 2024-09-13T09:54:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_40 2024-09-13T09:54:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_40 2024-09-13T09:54:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-13T09:54:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-13T09:54:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_41 2024-09-13T09:54:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-09-13T09:54:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:54:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_41 2024-09-13T09:54:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_41 2024-09-13T09:54:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-13T09:54:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-13T09:54:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-09-13T09:54:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_41 2024-09-13T09:54:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_41 2024-09-13T09:54:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_41 2024-09-13T09:54:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-13T09:54:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-13T09:54:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_42 2024-09-13T09:54:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-09-13T09:54:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:54:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_42 2024-09-13T09:54:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_42 2024-09-13T09:54:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-13T09:54:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-13T09:54:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-09-13T09:54:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_42 2024-09-13T09:54:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_42 2024-09-13T09:54:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_42 2024-09-13T09:54:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-13T09:54:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-13T09:54:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_43 2024-09-13T09:54:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-09-13T09:54:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:54:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_43 2024-09-13T09:54:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_43 2024-09-13T09:54:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-13T09:54:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-13T09:54:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-09-13T09:54:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_43 2024-09-13T09:54:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_43 2024-09-13T09:54:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_43 2024-09-13T09:54:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-13T09:54:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-13T09:54:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_44 2024-09-13T09:54:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-09-13T09:54:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:54:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_44 2024-09-13T09:54:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_44 2024-09-13T09:54:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-13T09:54:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-13T09:54:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-09-13T09:54:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_44 2024-09-13T09:54:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_44 2024-09-13T09:54:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_44 2024-09-13T09:54:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-13T09:54:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-13T09:54:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_45 2024-09-13T09:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-09-13T09:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_45 2024-09-13T09:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_45 2024-09-13T09:54:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-13T09:54:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-13T09:54:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-09-13T09:54:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_45 2024-09-13T09:54:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_45 2024-09-13T09:54:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_45 2024-09-13T09:54:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-13T09:54:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-13T09:54:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_46 2024-09-13T09:54:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-09-13T09:54:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:54:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_46 2024-09-13T09:54:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_46 2024-09-13T09:54:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-13T09:54:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-13T09:54:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-09-13T09:54:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_46 2024-09-13T09:54:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_46 2024-09-13T09:54:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_46 2024-09-13T09:54:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-13T09:54:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-13T09:54:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_47 2024-09-13T09:54:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-09-13T09:54:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:54:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_47 2024-09-13T09:54:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_47 2024-09-13T09:54:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-13T09:54:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-13T09:54:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-09-13T09:54:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_47 2024-09-13T09:54:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_47 2024-09-13T09:54:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_47 2024-09-13T09:54:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-13T09:54:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:44.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-13T09:54:44.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:44.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:44.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:44.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:44.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:44.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_48 2024-09-13T09:54:44.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-09-13T09:54:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:54:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_48 2024-09-13T09:54:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_48 2024-09-13T09:54:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-13T09:54:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-13T09:54:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-09-13T09:54:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_48 2024-09-13T09:54:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_48 2024-09-13T09:54:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_48 2024-09-13T09:54:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-13T09:54:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-13T09:54:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_49 2024-09-13T09:54:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-09-13T09:54:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:54:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_49 2024-09-13T09:54:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_49 2024-09-13T09:54:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-13T09:54:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-13T09:54:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-09-13T09:54:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_49 2024-09-13T09:54:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_49 2024-09-13T09:54:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_49 2024-09-13T09:54:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-13T09:54:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-13T09:54:45.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-13T09:54:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_50 2024-09-13T09:54:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-09-13T09:54:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:54:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:54:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:54:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_50 2024-09-13T09:54:45.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:45.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:45.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_50 2024-09-13T09:54:45.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-13T09:54:45.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:45.070 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-13T09:54:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:54:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:54:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:54:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-09-13T09:54:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_50 2024-09-13T09:54:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:54:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:54:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:54:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:54:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_50 2024-09-13T09:54:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:54:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:54:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_50 2024-09-13T09:54:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-13T09:54:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:54:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-13T09:54:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:54:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:54:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:54:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:54:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-09-13T09:54:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee -a /tmp/tmp.V7YewHEkvl/rbd-mirror.test.log 2024-09-13T09:54:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ date '+%F %T' 2024-09-13T09:54:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo 2024-09-13 09:54:45 TEST: delete images during bootstrap 2024-09-13T09:54:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:2024-09-13 09:54:45 TEST: delete images during bootstrap 2024-09-13T09:54:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-09-13T09:54:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=image 2024-09-13T09:54:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-09-13T09:54:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd: mirroring is already configured for image mode 2024-09-13T09:54:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-09-13T09:54:45.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:45.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:45.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=image 2024-09-13T09:54:45.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-09-13T09:54:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd: mirroring is already configured for image mode 2024-09-13T09:54:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ start_mirror cluster1 2024-09-13T09:54:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:54:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance 2024-09-13T09:54:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set_cluster_instance cluster1 cluster instance 2024-09-13T09:54:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local val=cluster1 2024-09-13T09:54:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster_var_name=cluster 2024-09-13T09:54:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local instance_var_name=instance 2024-09-13T09:54:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cluster=cluster1 2024-09-13T09:54:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=cluster1 2024-09-13T09:54:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' cluster1 = cluster1 ']' 2024-09-13T09:54:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ instance=0 2024-09-13T09:54:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval cluster=cluster1 2024-09-13T09:54:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ cluster=cluster1 2024-09-13T09:54:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ eval instance=0 2024-09-13T09:54:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ instance=0 2024-09-13T09:54:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2024-09-13T09:54:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2024-09-13T09:54:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=test 2024-09-13T09:54:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ seq 1 10 2024-09-13T09:54:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:54:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_1 2024-09-13T09:54:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_1 512M 2024-09-13T09:54:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:54:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:54:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:54:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:54:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:54:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 512M ']' 2024-09-13T09:54:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2024-09-13T09:54:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:54:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_1 2024-09-13T09:54:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_1 2024-09-13T09:54:45.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:45.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:45.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:54:45.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:54:45.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-13T09:54:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:54:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-13T09:54:46.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_1': 2024-09-13T09:54:46.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T09:54:46.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:54:46.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:54:46.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 190dd0bd211b 2024-09-13T09:54:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.190dd0bd211b 2024-09-13T09:54:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:54:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:54:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:54:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:54:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:54:45 2024 2024-09-13T09:54:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:54:45 2024 2024-09-13T09:54:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:54:45 2024 2024-09-13T09:54:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:54:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:54:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: ded3ac92-1108-4023-9876-0788572e0749 2024-09-13T09:54:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:54:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-13T09:54:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:54:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:54:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:54:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:54:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=14 2024-09-13T09:54:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:54:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:54:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 90494976 bytes 2024-09-13T09:54:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 91140096 bytes 2024-09-13T09:54:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 107816960 bytes 2024-09-13T09:54:46.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 126838784 bytes 2024-09-13T09:54:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 223349760 bytes 2024-09-13T09:54:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 257081344 bytes 2024-09-13T09:54:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 309411840 bytes 2024-09-13T09:54:47.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 334252032 bytes 2024-09-13T09:54:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 396648448 bytes 2024-09-13T09:54:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 397264896 bytes 2024-09-13T09:54:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 440764416 bytes 2024-09-13T09:54:47.666 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6979994333373402) > weight (0.25) 2024-09-13T09:54:47.666 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7155999007577878) > weight (0.25) 2024-09-13T09:54:47.666 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9059814943384282) > weight (0.25) 2024-09-13T09:54:47.666 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7284939394834647) > weight (0.25) 2024-09-13T09:54:47.666 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2024-09-13T09:54:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 466851840 bytes 2024-09-13T09:54:47.780 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 486634496 bytes 2024-09-13T09:54:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 552758272 bytes 2024-09-13T09:54:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 553692160 bytes 2024-09-13T09:54:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 607112192 bytes 2024-09-13T09:54:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 676941824 bytes 2024-09-13T09:54:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 703088640 bytes 2024-09-13T09:54:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 744247296 bytes 2024-09-13T09:54:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 808373248 bytes 2024-09-13T09:54:48.783 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 854207488 bytes 2024-09-13T09:54:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 914104320 bytes 2024-09-13T09:54:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 968175616 bytes 2024-09-13T09:54:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1002526720 bytes 2024-09-13T09:54:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1028252672 bytes 2024-09-13T09:54:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1089011712 bytes 2024-09-13T09:54:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1155241984 bytes 2024-09-13T09:54:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1177933824 bytes 2024-09-13T09:54:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1212791808 bytes 2024-09-13T09:54:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1261194240 bytes 2024-09-13T09:54:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1273812992 bytes 2024-09-13T09:54:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1334966272 bytes 2024-09-13T09:54:50.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1375813632 bytes 2024-09-13T09:54:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1394256896 bytes 2024-09-13T09:54:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1406642176 bytes 2024-09-13T09:54:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1455340544 bytes 2024-09-13T09:54:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1459833856 bytes 2024-09-13T09:54:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1513466880 bytes 2024-09-13T09:54:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1514655744 bytes 2024-09-13T09:54:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1528388608 bytes 2024-09-13T09:54:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1546474496 bytes 2024-09-13T09:54:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1560263680 bytes 2024-09-13T09:54:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1590032384 bytes 2024-09-13T09:54:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1615068160 bytes 2024-09-13T09:54:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1725870080 bytes 2024-09-13T09:54:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1801431040 bytes 2024-09-13T09:54:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1837662208 bytes 2024-09-13T09:54:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1869765632 bytes 2024-09-13T09:54:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1888983040 bytes 2024-09-13T09:54:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1923387392 bytes 2024-09-13T09:54:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 1964098560 bytes 2024-09-13T09:54:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 1964710912 bytes 2024-09-13T09:54:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 1979132928 bytes 2024-09-13T09:54:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2073463808 bytes 2024-09-13T09:54:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2153938944 bytes 2024-09-13T09:54:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2188274688 bytes 2024-09-13T09:54:53.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2213862400 bytes 2024-09-13T09:54:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2250068992 bytes 2024-09-13T09:54:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2303519744 bytes 2024-09-13T09:54:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2343675904 bytes 2024-09-13T09:54:53.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2386155520 bytes 2024-09-13T09:54:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2421169152 bytes 2024-09-13T09:54:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2497439744 bytes 2024-09-13T09:54:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2526209024 bytes 2024-09-13T09:54:54.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2612790272 bytes 2024-09-13T09:54:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2671616000 bytes 2024-09-13T09:54:54.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2677058560 bytes 2024-09-13T09:54:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2705696768 bytes 2024-09-13T09:54:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2766793728 bytes 2024-09-13T09:54:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2809315328 bytes 2024-09-13T09:54:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2826610688 bytes 2024-09-13T09:54:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2896788480 bytes 2024-09-13T09:54:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2926149632 bytes 2024-09-13T09:54:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 2962253824 bytes 2024-09-13T09:54:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3019059200 bytes 2024-09-13T09:54:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3101598720 bytes 2024-09-13T09:54:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3185817600 bytes 2024-09-13T09:54:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3202637824 bytes 2024-09-13T09:54:55.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3248571392 bytes 2024-09-13T09:54:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3313557504 bytes 2024-09-13T09:54:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3321496576 bytes 2024-09-13T09:54:56.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3371876352 bytes 2024-09-13T09:54:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3450561536 bytes 2024-09-13T09:54:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3503364096 bytes 2024-09-13T09:54:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3539149824 bytes 2024-09-13T09:54:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3576837120 bytes 2024-09-13T09:54:56.951 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3617159168 bytes 2024-09-13T09:54:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3669588992 bytes 2024-09-13T09:54:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3703215104 bytes 2024-09-13T09:54:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 3780793344 bytes 2024-09-13T09:54:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 3806070784 bytes 2024-09-13T09:54:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 3862544384 bytes 2024-09-13T09:54:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 3866828800 bytes 2024-09-13T09:54:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9400 IOs, 3922183168 bytes 2024-09-13T09:54:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9500 IOs, 4005073920 bytes 2024-09-13T09:54:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9600 IOs, 4041574400 bytes 2024-09-13T09:54:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9700 IOs, 4101251072 bytes 2024-09-13T09:54:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9800 IOs, 4185071616 bytes 2024-09-13T09:54:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9900 IOs, 4194906112 bytes 2024-09-13T09:54:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10000 IOs, 4320972800 bytes 2024-09-13T09:54:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10100 IOs, 4329167872 bytes 2024-09-13T09:54:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10200 IOs, 4359438336 bytes 2024-09-13T09:54:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10300 IOs, 4404371456 bytes 2024-09-13T09:54:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10400 IOs, 4404981760 bytes 2024-09-13T09:54:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10500 IOs, 4480139264 bytes 2024-09-13T09:54:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10600 IOs, 4554159104 bytes 2024-09-13T09:54:59.545 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10700 IOs, 4567100416 bytes 2024-09-13T09:54:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10800 IOs, 4638211072 bytes 2024-09-13T09:54:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10900 IOs, 4670440448 bytes 2024-09-13T09:54:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11000 IOs, 4707030016 bytes 2024-09-13T09:55:00.033 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11100 IOs, 4716615680 bytes 2024-09-13T09:55:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11200 IOs, 4815777792 bytes 2024-09-13T09:55:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:55:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:55:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:55:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:55:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-09-13T09:55:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:55:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:55:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:55:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=present 2024-09-13T09:55:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:55:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:55:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:55:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_1 2024-09-13T09:55:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:55:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:55:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_1 2024-09-13T09:55:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-13T09:55:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:55:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=1282bebccdf3 2024-09-13T09:55:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:55:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:55:00.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_1 present 1282bebccdf3 2024-09-13T09:55:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:55:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:55:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:55:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=present 2024-09-13T09:55:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=1282bebccdf3 2024-09-13T09:55:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:55:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:55:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_1 2024-09-13T09:55:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:55:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:55:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_1 2024-09-13T09:55:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-13T09:55:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:55:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=1282bebccdf3 2024-09-13T09:55:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1282bebccdf3 2024-09-13T09:55:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 1282bebccdf3 -o 1282bebccdf3 = 1282bebccdf3 2024-09-13T09:55:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2024-09-13T09:55:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test present = present 2024-09-13T09:55:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:55:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ disable_mirror cluster2 mirror image_1 2024-09-13T09:55:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:55:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:55:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:55:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-09-13T09:55:01.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T09:55:01.227+0000 7f7171dbb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f303dd0000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=368 2024-09-13T09:55:01.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T09:55:01.227+0000 7f7171dbb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f303dd0000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=369, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b8d40c1b-48a7-4e9b-8cbe-df65c34733d8, primary_snap_id=170, last_copied_object_number=128, snap_seqs={368=18446744073709551614}] 2024-09-13T09:55:01.454 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2972635416332171) > weight (0.25) 2024-09-13T09:55:01.454 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7862490519323017) > weight (0.25) 2024-09-13T09:55:01.454 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-13T09:55:01.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-13T09:55:01.455 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-13T09:55:01.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T09:55:01.455+0000 7f67289d7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 51563) UID: 1000 2024-09-13T09:55:01.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T09:55:01.749+0000 7fb8e3b8e640 -1 rbd::mirror::ImageMap: 0x55594d33e900 handle_peer_ack_remove: failed to remove global_image_id=ded3ac92-1108-4023-9876-0788572e0749 2024-09-13T09:55:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring disabled 2024-09-13T09:55:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-13T09:55:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:55:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:55:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:55:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:55:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:55:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:55:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:55:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_1 2024-09-13T09:55:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:55:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:55:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_1 2024-09-13T09:55:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-13T09:55:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:55:06.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-13T09:55:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:55:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:55:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:55:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-13T09:55:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:55:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:55:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:55:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:55:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:55:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:55:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:55:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_1 2024-09-13T09:55:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:55:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:55:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_1 2024-09-13T09:55:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-13T09:55:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:55:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-13T09:55:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:55:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:55:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:55:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:55:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ purge_snapshots cluster2 mirror image_1 2024-09-13T09:55:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:55:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:55:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:55:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-09-13T09:55:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-13T09:55:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:55:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:55:06.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:55:06.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:55:06.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:55:06.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_1 2024-09-13T09:55:06.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:55:06.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:55:06.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_1 2024-09-13T09:55:06.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-13T09:55:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-13T09:55:08.456 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-13T09:55:08.456 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:55:08.456 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-13T09:55:08.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-13T09:55:08.457 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-13T09:55:08.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-13T09:55:08.460 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2024-09-13T09:55:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-13T09:55:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:55:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:55:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_2 2024-09-13T09:55:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_2 512M 2024-09-13T09:55:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:55:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:55:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:55:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:55:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:55:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:55:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:55:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 512M ']' 2024-09-13T09:55:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2024-09-13T09:55:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:55:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_2 2024-09-13T09:55:29.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_2 2024-09-13T09:55:29.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:55:29.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:55:29.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:55:29.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:55:29.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-13T09:55:30.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:55:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-13T09:55:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_2': 2024-09-13T09:55:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T09:55:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:55:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:55:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 192e45a641fd 2024-09-13T09:55:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.192e45a641fd 2024-09-13T09:55:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:55:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:55:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:55:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:55:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:55:29 2024 2024-09-13T09:55:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:55:29 2024 2024-09-13T09:55:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:55:29 2024 2024-09-13T09:55:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:55:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:55:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 9049158c-2504-45b2-b4d4-e43ab9445b0d 2024-09-13T09:55:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:55:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-13T09:55:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:55:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:55:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:55:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:55:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=14 2024-09-13T09:55:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:55:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:55:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 42411008 bytes 2024-09-13T09:55:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 77522944 bytes 2024-09-13T09:55:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 113093632 bytes 2024-09-13T09:55:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 155554816 bytes 2024-09-13T09:55:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 190160896 bytes 2024-09-13T09:55:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 252221440 bytes 2024-09-13T09:55:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 311235584 bytes 2024-09-13T09:55:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 361645056 bytes 2024-09-13T09:55:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 369462272 bytes 2024-09-13T09:55:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 460491776 bytes 2024-09-13T09:55:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 486854656 bytes 2024-09-13T09:55:31.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 531575808 bytes 2024-09-13T09:55:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 579778560 bytes 2024-09-13T09:55:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 580396032 bytes 2024-09-13T09:55:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 663106560 bytes 2024-09-13T09:55:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 702635008 bytes 2024-09-13T09:55:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 719143936 bytes 2024-09-13T09:55:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 727990272 bytes 2024-09-13T09:55:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 756785152 bytes 2024-09-13T09:55:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 779734016 bytes 2024-09-13T09:55:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 818520064 bytes 2024-09-13T09:55:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 861295616 bytes 2024-09-13T09:55:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 897698816 bytes 2024-09-13T09:55:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 960245760 bytes 2024-09-13T09:55:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 990536704 bytes 2024-09-13T09:55:33.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1060656128 bytes 2024-09-13T09:55:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1085308928 bytes 2024-09-13T09:55:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1136632832 bytes 2024-09-13T09:55:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1189428224 bytes 2024-09-13T09:55:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1230920704 bytes 2024-09-13T09:55:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1242397696 bytes 2024-09-13T09:55:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1319416832 bytes 2024-09-13T09:55:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1390789632 bytes 2024-09-13T09:55:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1445403648 bytes 2024-09-13T09:55:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1446052864 bytes 2024-09-13T09:55:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1477870592 bytes 2024-09-13T09:55:34.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1576096768 bytes 2024-09-13T09:55:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1666057216 bytes 2024-09-13T09:55:35.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1732980736 bytes 2024-09-13T09:55:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1797730304 bytes 2024-09-13T09:55:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1817123840 bytes 2024-09-13T09:55:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1824510976 bytes 2024-09-13T09:55:35.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1859816448 bytes 2024-09-13T09:55:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1880925184 bytes 2024-09-13T09:55:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1947515904 bytes 2024-09-13T09:55:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1975687168 bytes 2024-09-13T09:55:35.998 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2012201984 bytes 2024-09-13T09:55:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2088062976 bytes 2024-09-13T09:55:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2102512640 bytes 2024-09-13T09:55:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2140065792 bytes 2024-09-13T09:55:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2140680192 bytes 2024-09-13T09:55:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2165526528 bytes 2024-09-13T09:55:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2166167552 bytes 2024-09-13T09:55:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2186740736 bytes 2024-09-13T09:55:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2265146368 bytes 2024-09-13T09:55:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2286544896 bytes 2024-09-13T09:55:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2287155200 bytes 2024-09-13T09:55:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2323545088 bytes 2024-09-13T09:55:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2381814784 bytes 2024-09-13T09:55:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2441231360 bytes 2024-09-13T09:55:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2465428480 bytes 2024-09-13T09:55:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2505653248 bytes 2024-09-13T09:55:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2517050368 bytes 2024-09-13T09:55:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2541440000 bytes 2024-09-13T09:55:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2642759680 bytes 2024-09-13T09:55:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2668826624 bytes 2024-09-13T09:55:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2797367296 bytes 2024-09-13T09:55:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2904287232 bytes 2024-09-13T09:55:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2965809152 bytes 2024-09-13T09:55:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 3020172288 bytes 2024-09-13T09:55:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 3051853824 bytes 2024-09-13T09:55:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 3110872064 bytes 2024-09-13T09:55:39.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 3127730176 bytes 2024-09-13T09:55:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3246205952 bytes 2024-09-13T09:55:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3276797952 bytes 2024-09-13T09:55:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3376316416 bytes 2024-09-13T09:55:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3421593600 bytes 2024-09-13T09:55:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3459398656 bytes 2024-09-13T09:55:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3516796928 bytes 2024-09-13T09:55:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3596291072 bytes 2024-09-13T09:55:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3599645696 bytes 2024-09-13T09:55:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3627567104 bytes 2024-09-13T09:55:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3727807488 bytes 2024-09-13T09:55:40.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3802340352 bytes 2024-09-13T09:55:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3845736448 bytes 2024-09-13T09:55:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3885835264 bytes 2024-09-13T09:55:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3922180096 bytes 2024-09-13T09:55:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3936519168 bytes 2024-09-13T09:55:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3994592256 bytes 2024-09-13T09:55:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 4034644992 bytes 2024-09-13T09:55:41.715 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 4065539072 bytes 2024-09-13T09:55:41.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 4158724096 bytes 2024-09-13T09:55:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 4187012096 bytes 2024-09-13T09:55:42.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9400 IOs, 4232762368 bytes 2024-09-13T09:55:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9500 IOs, 4278328320 bytes 2024-09-13T09:55:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9600 IOs, 4320661504 bytes 2024-09-13T09:55:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9700 IOs, 4352700416 bytes 2024-09-13T09:55:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9800 IOs, 4398612480 bytes 2024-09-13T09:55:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9900 IOs, 4400924672 bytes 2024-09-13T09:55:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10000 IOs, 4453006336 bytes 2024-09-13T09:55:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10100 IOs, 4504551424 bytes 2024-09-13T09:55:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10200 IOs, 4541007872 bytes 2024-09-13T09:55:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10300 IOs, 4584645632 bytes 2024-09-13T09:55:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10400 IOs, 4645677056 bytes 2024-09-13T09:55:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10500 IOs, 4647462912 bytes 2024-09-13T09:55:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10600 IOs, 4667543552 bytes 2024-09-13T09:55:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10700 IOs, 4692998144 bytes 2024-09-13T09:55:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10800 IOs, 4695229440 bytes 2024-09-13T09:55:43.893 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10900 IOs, 4727640064 bytes 2024-09-13T09:55:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11000 IOs, 4769302528 bytes 2024-09-13T09:55:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11100 IOs, 4832177152 bytes 2024-09-13T09:55:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11200 IOs, 4859937792 bytes 2024-09-13T09:55:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:55:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:55:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:55:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:55:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-09-13T09:55:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:55:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:55:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:55:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=present 2024-09-13T09:55:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:55:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:55:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:55:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_2 2024-09-13T09:55:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:55:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:55:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_2 2024-09-13T09:55:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-13T09:55:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:55:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=128232c9b264 2024-09-13T09:55:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:55:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:55:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_2 present 128232c9b264 2024-09-13T09:55:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:55:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:55:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:55:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=present 2024-09-13T09:55:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=128232c9b264 2024-09-13T09:55:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:55:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:55:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_2 2024-09-13T09:55:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:55:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:55:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_2 2024-09-13T09:55:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-13T09:55:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:55:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=128232c9b264 2024-09-13T09:55:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 128232c9b264 2024-09-13T09:55:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 128232c9b264 -o 128232c9b264 = 128232c9b264 2024-09-13T09:55:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2024-09-13T09:55:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test present = present 2024-09-13T09:55:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:55:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ disable_mirror cluster2 mirror image_2 2024-09-13T09:55:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:55:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:55:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:55:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-09-13T09:55:45.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T09:55:45.289+0000 7f7171dbb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f303f6e400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=370 2024-09-13T09:55:45.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T09:55:45.289+0000 7f7171dbb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f303f6e400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=371, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b8d40c1b-48a7-4e9b-8cbe-df65c34733d8, primary_snap_id=172, last_copied_object_number=128, snap_seqs={370=18446744073709551614}] 2024-09-13T09:55:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring disabled 2024-09-13T09:55:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-13T09:55:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:55:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:55:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:55:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:55:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:55:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:55:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:55:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_2 2024-09-13T09:55:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:55:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:55:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_2 2024-09-13T09:55:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-13T09:55:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:55:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-13T09:55:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:55:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:55:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:55:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-13T09:55:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:55:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:55:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:55:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:55:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:55:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:55:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:55:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_2 2024-09-13T09:55:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:55:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:55:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_2 2024-09-13T09:55:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-13T09:55:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:55:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-13T09:55:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:55:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:55:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:55:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:55:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ purge_snapshots cluster2 mirror image_2 2024-09-13T09:55:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:55:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:55:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:55:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-09-13T09:55:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-13T09:55:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:55:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:55:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:55:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:55:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:55:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_2 2024-09-13T09:55:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:55:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:55:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_2 2024-09-13T09:55:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-13T09:55:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-13T09:55:52.667 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5315080049544425) > weight (0.25) 2024-09-13T09:55:52.667 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3204970352828741) > weight (0.25) 2024-09-13T09:55:52.667 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.42046709591851816) > weight (0.25) 2024-09-13T09:55:52.667 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.34812612996645154) > weight (0.25) 2024-09-13T09:55:52.668 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2024-09-13T09:56:10.461 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-13T09:56:10.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-13T09:56:10.463 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-13T09:56:10.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:56:10.462+0000 7efcc5efa640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 60334) UID: 1000 2024-09-13T09:56:11.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-13T09:56:11.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:56:11.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:56:11.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_3 2024-09-13T09:56:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_3 512M 2024-09-13T09:56:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:56:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:56:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:56:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:56:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:56:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:56:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:56:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 512M ']' 2024-09-13T09:56:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2024-09-13T09:56:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:56:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_3 2024-09-13T09:56:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_3 2024-09-13T09:56:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:56:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:56:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:56:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:56:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-13T09:56:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:56:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-13T09:56:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_3': 2024-09-13T09:56:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T09:56:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:56:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:56:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 194129705375 2024-09-13T09:56:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.194129705375 2024-09-13T09:56:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:56:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:56:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:56:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:56:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:56:11 2024 2024-09-13T09:56:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:56:11 2024 2024-09-13T09:56:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:56:11 2024 2024-09-13T09:56:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:56:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:56:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4968e127-be40-4d38-8bf2-17444f098baa 2024-09-13T09:56:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:56:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-13T09:56:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:56:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:56:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:56:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:56:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=13 2024-09-13T09:56:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:56:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:56:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 25351168 bytes 2024-09-13T09:56:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 70218752 bytes 2024-09-13T09:56:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 123922432 bytes 2024-09-13T09:56:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 159010816 bytes 2024-09-13T09:56:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 207925248 bytes 2024-09-13T09:56:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 243565568 bytes 2024-09-13T09:56:13.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 285357056 bytes 2024-09-13T09:56:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 348990464 bytes 2024-09-13T09:56:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 393299968 bytes 2024-09-13T09:56:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 393931776 bytes 2024-09-13T09:56:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 440287232 bytes 2024-09-13T09:56:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 468826112 bytes 2024-09-13T09:56:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 497633280 bytes 2024-09-13T09:56:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 529844224 bytes 2024-09-13T09:56:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 551896064 bytes 2024-09-13T09:56:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 577069056 bytes 2024-09-13T09:56:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 626279424 bytes 2024-09-13T09:56:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 649100288 bytes 2024-09-13T09:56:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 704342016 bytes 2024-09-13T09:56:14.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 704972800 bytes 2024-09-13T09:56:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 741643264 bytes 2024-09-13T09:56:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 804247552 bytes 2024-09-13T09:56:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 889522176 bytes 2024-09-13T09:56:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 917213184 bytes 2024-09-13T09:56:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 951337984 bytes 2024-09-13T09:56:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 995993600 bytes 2024-09-13T09:56:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1069138944 bytes 2024-09-13T09:56:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1116509184 bytes 2024-09-13T09:56:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1179500544 bytes 2024-09-13T09:56:15.830 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1196968960 bytes 2024-09-13T09:56:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1209824256 bytes 2024-09-13T09:56:16.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1211633664 bytes 2024-09-13T09:56:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1242673152 bytes 2024-09-13T09:56:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1279782912 bytes 2024-09-13T09:56:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1329920000 bytes 2024-09-13T09:56:16.464 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-13T09:56:16.465 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:56:16.465 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-13T09:56:16.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-13T09:56:16.465 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-13T09:56:16.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-13T09:56:16.470 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2024-09-13T09:56:16.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1375315968 bytes 2024-09-13T09:56:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1447921664 bytes 2024-09-13T09:56:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1481733120 bytes 2024-09-13T09:56:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1555798016 bytes 2024-09-13T09:56:17.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1570447360 bytes 2024-09-13T09:56:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1626731520 bytes 2024-09-13T09:56:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1690001408 bytes 2024-09-13T09:56:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1746013184 bytes 2024-09-13T09:56:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1754299392 bytes 2024-09-13T09:56:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1810915328 bytes 2024-09-13T09:56:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1870379008 bytes 2024-09-13T09:56:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1915240448 bytes 2024-09-13T09:56:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1973599232 bytes 2024-09-13T09:56:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2073148416 bytes 2024-09-13T09:56:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2076187648 bytes 2024-09-13T09:56:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2157917184 bytes 2024-09-13T09:56:18.703 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2197318656 bytes 2024-09-13T09:56:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2211273728 bytes 2024-09-13T09:56:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2248091648 bytes 2024-09-13T09:56:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2248691712 bytes 2024-09-13T09:56:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2296899584 bytes 2024-09-13T09:56:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2302889984 bytes 2024-09-13T09:56:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2322111488 bytes 2024-09-13T09:56:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2355113984 bytes 2024-09-13T09:56:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2375258112 bytes 2024-09-13T09:56:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2424504320 bytes 2024-09-13T09:56:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2490228736 bytes 2024-09-13T09:56:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2539173888 bytes 2024-09-13T09:56:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2615946240 bytes 2024-09-13T09:56:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2616554496 bytes 2024-09-13T09:56:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2665184256 bytes 2024-09-13T09:56:20.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2751036416 bytes 2024-09-13T09:56:20.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2802977792 bytes 2024-09-13T09:56:20.692 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2827770880 bytes 2024-09-13T09:56:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2854126592 bytes 2024-09-13T09:56:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2871876608 bytes 2024-09-13T09:56:21.050 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2948686848 bytes 2024-09-13T09:56:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2971067392 bytes 2024-09-13T09:56:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3016768512 bytes 2024-09-13T09:56:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3052341248 bytes 2024-09-13T09:56:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3080109056 bytes 2024-09-13T09:56:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3100580864 bytes 2024-09-13T09:56:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3138661376 bytes 2024-09-13T09:56:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3170126848 bytes 2024-09-13T09:56:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3204626432 bytes 2024-09-13T09:56:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3266519040 bytes 2024-09-13T09:56:22.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3276789760 bytes 2024-09-13T09:56:22.316 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3310164992 bytes 2024-09-13T09:56:22.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3398902784 bytes 2024-09-13T09:56:22.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3409766400 bytes 2024-09-13T09:56:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3449606144 bytes 2024-09-13T09:56:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3488098304 bytes 2024-09-13T09:56:22.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3522119680 bytes 2024-09-13T09:56:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3591644160 bytes 2024-09-13T09:56:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 3653643264 bytes 2024-09-13T09:56:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 3737399296 bytes 2024-09-13T09:56:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 3795178496 bytes 2024-09-13T09:56:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 3871378432 bytes 2024-09-13T09:56:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9400 IOs, 3871992832 bytes 2024-09-13T09:56:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9500 IOs, 3928043520 bytes 2024-09-13T09:56:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9600 IOs, 3962191872 bytes 2024-09-13T09:56:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9700 IOs, 4017831936 bytes 2024-09-13T09:56:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9800 IOs, 4075492352 bytes 2024-09-13T09:56:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9900 IOs, 4147341312 bytes 2024-09-13T09:56:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10000 IOs, 4170425344 bytes 2024-09-13T09:56:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10100 IOs, 4190408704 bytes 2024-09-13T09:56:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10200 IOs, 4194066432 bytes 2024-09-13T09:56:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10300 IOs, 4204293120 bytes 2024-09-13T09:56:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10400 IOs, 4220441600 bytes 2024-09-13T09:56:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10500 IOs, 4260271104 bytes 2024-09-13T09:56:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10600 IOs, 4348098560 bytes 2024-09-13T09:56:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:56:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:56:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:56:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:56:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-09-13T09:56:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:56:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:56:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:56:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=present 2024-09-13T09:56:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:56:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:56:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:56:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_3 2024-09-13T09:56:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:56:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:56:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_3 2024-09-13T09:56:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-13T09:56:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:56:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=1282de62e755 2024-09-13T09:56:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:56:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:56:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_3 present 1282de62e755 2024-09-13T09:56:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:56:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:56:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:56:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=present 2024-09-13T09:56:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=1282de62e755 2024-09-13T09:56:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:56:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:56:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_3 2024-09-13T09:56:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:56:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:56:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_3 2024-09-13T09:56:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-13T09:56:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:56:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=1282de62e755 2024-09-13T09:56:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1282de62e755 2024-09-13T09:56:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 1282de62e755 -o 1282de62e755 = 1282de62e755 2024-09-13T09:56:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2024-09-13T09:56:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test present = present 2024-09-13T09:56:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:56:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ disable_mirror cluster2 mirror image_3 2024-09-13T09:56:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:56:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:56:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:56:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-09-13T09:56:26.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T09:56:26.240+0000 7f7171dbb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f303c4ac00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=372 2024-09-13T09:56:26.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T09:56:26.240+0000 7f7171dbb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f303c4ac00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=373, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b8d40c1b-48a7-4e9b-8cbe-df65c34733d8, primary_snap_id=174, last_copied_object_number=128, snap_seqs={372=18446744073709551614}] 2024-09-13T09:56:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring disabled 2024-09-13T09:56:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-13T09:56:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:56:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:56:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:56:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:56:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:56:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:56:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:56:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_3 2024-09-13T09:56:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:56:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:56:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_3 2024-09-13T09:56:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-13T09:56:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:56:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-13T09:56:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:56:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:56:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:56:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-13T09:56:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:56:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:56:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:56:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:56:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:56:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:56:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:56:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_3 2024-09-13T09:56:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:56:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:56:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_3 2024-09-13T09:56:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-13T09:56:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:56:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-13T09:56:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:56:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:56:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:56:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:56:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ purge_snapshots cluster2 mirror image_3 2024-09-13T09:56:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:56:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:56:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:56:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-09-13T09:56:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-13T09:56:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:56:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:56:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:56:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:56:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:56:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_3 2024-09-13T09:56:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:56:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:56:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_3 2024-09-13T09:56:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-13T09:56:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-13T09:56:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-13T09:56:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:56:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:56:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_4 2024-09-13T09:56:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_4 512M 2024-09-13T09:56:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:56:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:56:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:56:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:56:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:56:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:56:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:56:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 512M ']' 2024-09-13T09:56:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2024-09-13T09:56:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:56:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_4 2024-09-13T09:56:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_4 2024-09-13T09:56:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:56:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:56:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:56:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:56:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-13T09:56:54.274 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:56:54.282 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-13T09:56:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_4': 2024-09-13T09:56:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T09:56:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:56:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:56:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 195c485774ef 2024-09-13T09:56:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.195c485774ef 2024-09-13T09:56:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:56:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:56:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:56:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:56:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:56:53 2024 2024-09-13T09:56:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:56:53 2024 2024-09-13T09:56:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:56:53 2024 2024-09-13T09:56:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:56:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:56:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 4b0d1200-3620-4b70-b56a-3f15876c829a 2024-09-13T09:56:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:56:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-13T09:56:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:56:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:56:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:56:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:56:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=14 2024-09-13T09:56:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:56:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:56:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 53264384 bytes 2024-09-13T09:56:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 84120576 bytes 2024-09-13T09:56:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 84739072 bytes 2024-09-13T09:56:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 87635968 bytes 2024-09-13T09:56:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 120072192 bytes 2024-09-13T09:56:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 204986368 bytes 2024-09-13T09:56:55.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 236572672 bytes 2024-09-13T09:56:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 286255104 bytes 2024-09-13T09:56:55.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 319519744 bytes 2024-09-13T09:56:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 374122496 bytes 2024-09-13T09:56:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 474571776 bytes 2024-09-13T09:56:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 538630144 bytes 2024-09-13T09:56:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 589302784 bytes 2024-09-13T09:56:56.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 618746880 bytes 2024-09-13T09:56:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 636517376 bytes 2024-09-13T09:56:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 691397632 bytes 2024-09-13T09:56:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 740629504 bytes 2024-09-13T09:56:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 792765440 bytes 2024-09-13T09:56:56.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 847730688 bytes 2024-09-13T09:56:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 869112832 bytes 2024-09-13T09:56:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 917739520 bytes 2024-09-13T09:56:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1023166464 bytes 2024-09-13T09:56:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1059392512 bytes 2024-09-13T09:56:57.433 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1082400768 bytes 2024-09-13T09:56:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1104239616 bytes 2024-09-13T09:56:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1113815040 bytes 2024-09-13T09:56:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1160745984 bytes 2024-09-13T09:56:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1216252928 bytes 2024-09-13T09:56:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1247694848 bytes 2024-09-13T09:56:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1288934400 bytes 2024-09-13T09:56:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1336969216 bytes 2024-09-13T09:56:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1396096000 bytes 2024-09-13T09:56:58.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1396721664 bytes 2024-09-13T09:56:58.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1400662016 bytes 2024-09-13T09:56:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1441816576 bytes 2024-09-13T09:56:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1442422784 bytes 2024-09-13T09:56:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1487430656 bytes 2024-09-13T09:56:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1563569152 bytes 2024-09-13T09:56:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1683301376 bytes 2024-09-13T09:56:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1698760704 bytes 2024-09-13T09:56:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1748599808 bytes 2024-09-13T09:56:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1759828992 bytes 2024-09-13T09:56:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1785720832 bytes 2024-09-13T09:56:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1824270336 bytes 2024-09-13T09:56:59.837 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1854114816 bytes 2024-09-13T09:56:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1942416384 bytes 2024-09-13T09:57:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1961477120 bytes 2024-09-13T09:57:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2025194496 bytes 2024-09-13T09:57:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2039140352 bytes 2024-09-13T09:57:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2064973824 bytes 2024-09-13T09:57:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2149095424 bytes 2024-09-13T09:57:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2174493696 bytes 2024-09-13T09:57:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2232944640 bytes 2024-09-13T09:57:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2266322944 bytes 2024-09-13T09:57:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2296671232 bytes 2024-09-13T09:57:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2383882240 bytes 2024-09-13T09:57:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2460558336 bytes 2024-09-13T09:57:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2464600064 bytes 2024-09-13T09:57:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2482404352 bytes 2024-09-13T09:57:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2557671424 bytes 2024-09-13T09:57:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2578671616 bytes 2024-09-13T09:57:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2638480384 bytes 2024-09-13T09:57:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2639098880 bytes 2024-09-13T09:57:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2733214720 bytes 2024-09-13T09:57:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2764948480 bytes 2024-09-13T09:57:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2780641280 bytes 2024-09-13T09:57:02.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2796404736 bytes 2024-09-13T09:57:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2828017664 bytes 2024-09-13T09:57:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2856478720 bytes 2024-09-13T09:57:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2910017536 bytes 2024-09-13T09:57:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2946682880 bytes 2024-09-13T09:57:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2972190720 bytes 2024-09-13T09:57:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2991149056 bytes 2024-09-13T09:57:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3005458432 bytes 2024-09-13T09:57:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3054489600 bytes 2024-09-13T09:57:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3095259136 bytes 2024-09-13T09:57:03.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3114050560 bytes 2024-09-13T09:57:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3179003904 bytes 2024-09-13T09:57:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3186689024 bytes 2024-09-13T09:57:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3243682816 bytes 2024-09-13T09:57:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3301198848 bytes 2024-09-13T09:57:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3345057792 bytes 2024-09-13T09:57:04.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3407306752 bytes 2024-09-13T09:57:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3442375680 bytes 2024-09-13T09:57:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3486036992 bytes 2024-09-13T09:57:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3537319936 bytes 2024-09-13T09:57:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3573175296 bytes 2024-09-13T09:57:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3670381568 bytes 2024-09-13T09:57:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3717472256 bytes 2024-09-13T09:57:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 3720192000 bytes 2024-09-13T09:57:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 3822480384 bytes 2024-09-13T09:57:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 3838792704 bytes 2024-09-13T09:57:05.624 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 3869498368 bytes 2024-09-13T09:57:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9400 IOs, 3889276928 bytes 2024-09-13T09:57:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9500 IOs, 3889874944 bytes 2024-09-13T09:57:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9600 IOs, 3931324416 bytes 2024-09-13T09:57:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9700 IOs, 3948527616 bytes 2024-09-13T09:57:06.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9800 IOs, 3995670528 bytes 2024-09-13T09:57:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9900 IOs, 4045070336 bytes 2024-09-13T09:57:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10000 IOs, 4086156288 bytes 2024-09-13T09:57:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10100 IOs, 4147690496 bytes 2024-09-13T09:57:06.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10200 IOs, 4194743296 bytes 2024-09-13T09:57:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10300 IOs, 4265694208 bytes 2024-09-13T09:57:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10400 IOs, 4267976704 bytes 2024-09-13T09:57:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10500 IOs, 4338655232 bytes 2024-09-13T09:57:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10600 IOs, 4374515712 bytes 2024-09-13T09:57:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10700 IOs, 4449129472 bytes 2024-09-13T09:57:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10800 IOs, 4491991040 bytes 2024-09-13T09:57:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10900 IOs, 4513178624 bytes 2024-09-13T09:57:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11000 IOs, 4574885888 bytes 2024-09-13T09:57:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11100 IOs, 4626314240 bytes 2024-09-13T09:57:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11200 IOs, 4691565568 bytes 2024-09-13T09:57:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11300 IOs, 4713131008 bytes 2024-09-13T09:57:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11400 IOs, 4733870080 bytes 2024-09-13T09:57:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:11500 IOs, 4798142464 bytes 2024-09-13T09:57:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:57:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:57:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:57:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:57:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-09-13T09:57:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:57:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:57:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:57:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=present 2024-09-13T09:57:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:57:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:57:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:57:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_4 2024-09-13T09:57:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:57:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:57:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_4 2024-09-13T09:57:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-13T09:57:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:57:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=12824638a705 2024-09-13T09:57:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:57:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:57:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_4 present 12824638a705 2024-09-13T09:57:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:57:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:57:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:57:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=present 2024-09-13T09:57:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=12824638a705 2024-09-13T09:57:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:57:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:57:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_4 2024-09-13T09:57:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:57:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:57:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_4 2024-09-13T09:57:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-13T09:57:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:57:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=12824638a705 2024-09-13T09:57:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 12824638a705 2024-09-13T09:57:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 12824638a705 -o 12824638a705 = 12824638a705 2024-09-13T09:57:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2024-09-13T09:57:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test present = present 2024-09-13T09:57:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:57:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ disable_mirror cluster2 mirror image_4 2024-09-13T09:57:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:57:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:57:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:57:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-09-13T09:57:09.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T09:57:09.283+0000 7f71725bc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f301ee0800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=374 2024-09-13T09:57:09.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T09:57:09.283+0000 7f71725bc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f301ee0800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=375, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b8d40c1b-48a7-4e9b-8cbe-df65c34733d8, primary_snap_id=176, last_copied_object_number=128, snap_seqs={374=18446744073709551614}] 2024-09-13T09:57:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring disabled 2024-09-13T09:57:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-13T09:57:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:57:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:57:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:57:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:57:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:57:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:57:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:57:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_4 2024-09-13T09:57:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:57:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:57:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_4 2024-09-13T09:57:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-13T09:57:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:57:14.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-13T09:57:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:57:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:57:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:57:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-13T09:57:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:57:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:57:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:57:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:57:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:57:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:57:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:57:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_4 2024-09-13T09:57:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:57:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:57:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_4 2024-09-13T09:57:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-13T09:57:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:57:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-13T09:57:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:57:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:57:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:57:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:57:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ purge_snapshots cluster2 mirror image_4 2024-09-13T09:57:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:57:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:57:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:57:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-09-13T09:57:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-13T09:57:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:57:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:57:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:57:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:57:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:57:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_4 2024-09-13T09:57:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:57:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:57:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_4 2024-09-13T09:57:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-13T09:57:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-13T09:57:16.471 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.38415915172806425) > weight (0.25) 2024-09-13T09:57:16.471 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9198419590823764) > weight (0.25) 2024-09-13T09:57:16.471 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6401326273815978) > weight (0.25) 2024-09-13T09:57:16.471 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5106604644836908) > weight (0.25) 2024-09-13T09:57:16.472 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2024-09-13T09:57:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-13T09:57:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:57:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:57:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_5 2024-09-13T09:57:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_5 512M 2024-09-13T09:57:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:57:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:57:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:57:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:57:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:57:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:57:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:57:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 512M ']' 2024-09-13T09:57:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2024-09-13T09:57:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:57:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_5 2024-09-13T09:57:37.579 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_5 2024-09-13T09:57:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:57:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:57:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:57:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:57:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-13T09:57:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:57:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-13T09:57:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_5': 2024-09-13T09:57:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T09:57:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:57:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:57:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 196ec67c932b 2024-09-13T09:57:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.196ec67c932b 2024-09-13T09:57:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:57:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:57:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:57:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:57:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:57:37 2024 2024-09-13T09:57:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:57:37 2024 2024-09-13T09:57:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:57:37 2024 2024-09-13T09:57:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:57:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:57:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 1f258e65-2deb-4b1b-9028-0892d90bbed7 2024-09-13T09:57:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:57:38.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-13T09:57:38.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:57:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:57:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:57:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:57:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=13 2024-09-13T09:57:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:57:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:57:38.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 15822848 bytes 2024-09-13T09:57:38.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 69425152 bytes 2024-09-13T09:57:38.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 121326592 bytes 2024-09-13T09:57:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 159565824 bytes 2024-09-13T09:57:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 199321600 bytes 2024-09-13T09:57:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 230713344 bytes 2024-09-13T09:57:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 337032192 bytes 2024-09-13T09:57:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 474843136 bytes 2024-09-13T09:57:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 549542912 bytes 2024-09-13T09:57:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 594542592 bytes 2024-09-13T09:57:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 627343360 bytes 2024-09-13T09:57:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 634923008 bytes 2024-09-13T09:57:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 686182400 bytes 2024-09-13T09:57:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 725131264 bytes 2024-09-13T09:57:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 732533760 bytes 2024-09-13T09:57:40.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 798115840 bytes 2024-09-13T09:57:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 823820288 bytes 2024-09-13T09:57:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 852340736 bytes 2024-09-13T09:57:40.668 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-13T09:57:40.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-13T09:57:40.669 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-13T09:57:40.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T09:57:40.668+0000 7f86a0a93640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 50015) UID: 1000 2024-09-13T09:57:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 887838720 bytes 2024-09-13T09:57:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 898024448 bytes 2024-09-13T09:57:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 969613312 bytes 2024-09-13T09:57:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 979667968 bytes 2024-09-13T09:57:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1034729472 bytes 2024-09-13T09:57:41.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1055904768 bytes 2024-09-13T09:57:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1071304704 bytes 2024-09-13T09:57:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1100380160 bytes 2024-09-13T09:57:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1181805568 bytes 2024-09-13T09:57:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1247868928 bytes 2024-09-13T09:57:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1275779072 bytes 2024-09-13T09:57:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1312912384 bytes 2024-09-13T09:57:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1339623424 bytes 2024-09-13T09:57:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1364156416 bytes 2024-09-13T09:57:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1399599104 bytes 2024-09-13T09:57:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1479816192 bytes 2024-09-13T09:57:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1504839680 bytes 2024-09-13T09:57:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1517641728 bytes 2024-09-13T09:57:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1518267392 bytes 2024-09-13T09:57:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1551227904 bytes 2024-09-13T09:57:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1596774400 bytes 2024-09-13T09:57:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1651001344 bytes 2024-09-13T09:57:43.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1710412800 bytes 2024-09-13T09:57:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1718848512 bytes 2024-09-13T09:57:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1762452480 bytes 2024-09-13T09:57:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1776857088 bytes 2024-09-13T09:57:43.670 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-13T09:57:43.670 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:57:43.671 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-13T09:57:43.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-13T09:57:43.671 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-13T09:57:43.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-13T09:57:43.675 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-09-13T09:57:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1806252032 bytes 2024-09-13T09:57:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1847000064 bytes 2024-09-13T09:57:43.916 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1863239680 bytes 2024-09-13T09:57:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1911826432 bytes 2024-09-13T09:57:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1931783168 bytes 2024-09-13T09:57:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2017098752 bytes 2024-09-13T09:57:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2043920384 bytes 2024-09-13T09:57:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2049198080 bytes 2024-09-13T09:57:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2075476992 bytes 2024-09-13T09:57:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2164846592 bytes 2024-09-13T09:57:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2219902976 bytes 2024-09-13T09:57:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2266094592 bytes 2024-09-13T09:57:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2305065984 bytes 2024-09-13T09:57:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2330728448 bytes 2024-09-13T09:57:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2429808640 bytes 2024-09-13T09:57:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2478205952 bytes 2024-09-13T09:57:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2508961792 bytes 2024-09-13T09:57:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2565231616 bytes 2024-09-13T09:57:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2630593536 bytes 2024-09-13T09:57:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2686853120 bytes 2024-09-13T09:57:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2736192512 bytes 2024-09-13T09:57:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2783582208 bytes 2024-09-13T09:57:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2815009792 bytes 2024-09-13T09:57:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2848764928 bytes 2024-09-13T09:57:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2852170752 bytes 2024-09-13T09:57:46.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2904543232 bytes 2024-09-13T09:57:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2953135104 bytes 2024-09-13T09:57:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2996547584 bytes 2024-09-13T09:57:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 3072227328 bytes 2024-09-13T09:57:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3090767872 bytes 2024-09-13T09:57:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3125358592 bytes 2024-09-13T09:57:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3145529344 bytes 2024-09-13T09:57:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3229791232 bytes 2024-09-13T09:57:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3285107712 bytes 2024-09-13T09:57:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3321821184 bytes 2024-09-13T09:57:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3386134528 bytes 2024-09-13T09:57:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3471673344 bytes 2024-09-13T09:57:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3540073472 bytes 2024-09-13T09:57:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3588152320 bytes 2024-09-13T09:57:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3651513344 bytes 2024-09-13T09:57:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3652151296 bytes 2024-09-13T09:57:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3652742144 bytes 2024-09-13T09:57:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3693046784 bytes 2024-09-13T09:57:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3708746752 bytes 2024-09-13T09:57:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3723679744 bytes 2024-09-13T09:57:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 3749355520 bytes 2024-09-13T09:57:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 3749991424 bytes 2024-09-13T09:57:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 3750594560 bytes 2024-09-13T09:57:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 3794251776 bytes 2024-09-13T09:57:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9400 IOs, 3804695552 bytes 2024-09-13T09:57:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9500 IOs, 3857141760 bytes 2024-09-13T09:57:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9600 IOs, 3887201280 bytes 2024-09-13T09:57:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9700 IOs, 4013966336 bytes 2024-09-13T09:57:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9800 IOs, 4074244096 bytes 2024-09-13T09:57:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9900 IOs, 4126737408 bytes 2024-09-13T09:57:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10000 IOs, 4163680256 bytes 2024-09-13T09:57:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10100 IOs, 4257507328 bytes 2024-09-13T09:57:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10200 IOs, 4271266816 bytes 2024-09-13T09:57:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10300 IOs, 4296034304 bytes 2024-09-13T09:57:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10400 IOs, 4373201920 bytes 2024-09-13T09:57:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10500 IOs, 4414782464 bytes 2024-09-13T09:57:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10600 IOs, 4446723072 bytes 2024-09-13T09:57:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:57:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:57:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:57:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:57:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-09-13T09:57:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:57:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:57:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:57:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=present 2024-09-13T09:57:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:57:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:57:51.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:57:51.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_5 2024-09-13T09:57:51.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:57:51.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:57:51.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_5 2024-09-13T09:57:51.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-13T09:57:51.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:57:51.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=12821d932c4e 2024-09-13T09:57:51.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:57:51.441 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:57:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_5 present 12821d932c4e 2024-09-13T09:57:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:57:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:57:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:57:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=present 2024-09-13T09:57:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=12821d932c4e 2024-09-13T09:57:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:57:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:57:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_5 2024-09-13T09:57:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:57:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:57:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_5 2024-09-13T09:57:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-13T09:57:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:57:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=12821d932c4e 2024-09-13T09:57:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 12821d932c4e 2024-09-13T09:57:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 12821d932c4e -o 12821d932c4e = 12821d932c4e 2024-09-13T09:57:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2024-09-13T09:57:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test present = present 2024-09-13T09:57:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:57:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ disable_mirror cluster2 mirror image_5 2024-09-13T09:57:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:57:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:57:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:57:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-09-13T09:57:51.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T09:57:51.797+0000 7f7171dbb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f30120f800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=376 2024-09-13T09:57:51.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T09:57:51.797+0000 7f7171dbb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f30120f800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=377, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b8d40c1b-48a7-4e9b-8cbe-df65c34733d8, primary_snap_id=178, last_copied_object_number=128, snap_seqs={376=18446744073709551614}] 2024-09-13T09:57:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring disabled 2024-09-13T09:57:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-13T09:57:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:57:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:57:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:57:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:57:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:57:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:57:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:57:56.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_5 2024-09-13T09:57:56.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:57:56.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:57:56.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_5 2024-09-13T09:57:56.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-13T09:57:56.812 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:57:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-13T09:57:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:57:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:57:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:57:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-13T09:57:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:57:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:57:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:57:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:57:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:57:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:57:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:57:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_5 2024-09-13T09:57:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:57:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:57:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_5 2024-09-13T09:57:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-13T09:57:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:57:56.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-13T09:57:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:57:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:57:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:57:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:57:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ purge_snapshots cluster2 mirror image_5 2024-09-13T09:57:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:57:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:57:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:57:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-09-13T09:57:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-13T09:57:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:57:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:57:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:57:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:57:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:57:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_5 2024-09-13T09:57:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:57:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:57:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_5 2024-09-13T09:57:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-13T09:57:57.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-13T09:58:17.472 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8372475831118179) > weight (0.25) 2024-09-13T09:58:17.473 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5741226083464858) > weight (0.25) 2024-09-13T09:58:17.473 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3456925347926789) > weight (0.25) 2024-09-13T09:58:17.473 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5214299124309287) > weight (0.25) 2024-09-13T09:58:17.473 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-09-13T09:58:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-13T09:58:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:58:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:58:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_6 2024-09-13T09:58:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_6 512M 2024-09-13T09:58:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:58:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:58:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:58:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:58:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:58:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:58:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:58:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 512M ']' 2024-09-13T09:58:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2024-09-13T09:58:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:58:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_6 2024-09-13T09:58:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_6 2024-09-13T09:58:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:58:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:58:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:58:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:58:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-13T09:58:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:58:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-13T09:58:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_6': 2024-09-13T09:58:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T09:58:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:58:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:58:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 1986ec037d9a 2024-09-13T09:58:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.1986ec037d9a 2024-09-13T09:58:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:58:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:58:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:58:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:58:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:58:21 2024 2024-09-13T09:58:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:58:21 2024 2024-09-13T09:58:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:58:21 2024 2024-09-13T09:58:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:58:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:58:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 5bff9702-8d9d-4f58-a5bc-9dba352ec58c 2024-09-13T09:58:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:58:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-13T09:58:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:58:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:58:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:58:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:58:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=12 2024-09-13T09:58:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:58:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:58:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 47052800 bytes 2024-09-13T09:58:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 120890368 bytes 2024-09-13T09:58:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 145078272 bytes 2024-09-13T09:58:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 196176896 bytes 2024-09-13T09:58:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 254461952 bytes 2024-09-13T09:58:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 307477504 bytes 2024-09-13T09:58:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 324349952 bytes 2024-09-13T09:58:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 370525184 bytes 2024-09-13T09:58:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 462745600 bytes 2024-09-13T09:58:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 483087360 bytes 2024-09-13T09:58:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 543586304 bytes 2024-09-13T09:58:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 580746240 bytes 2024-09-13T09:58:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 620361728 bytes 2024-09-13T09:58:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 642320384 bytes 2024-09-13T09:58:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 647158784 bytes 2024-09-13T09:58:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 697353216 bytes 2024-09-13T09:58:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 733839360 bytes 2024-09-13T09:58:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 795956224 bytes 2024-09-13T09:58:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 832422912 bytes 2024-09-13T09:58:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 909836288 bytes 2024-09-13T09:58:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 946447360 bytes 2024-09-13T09:58:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 981849088 bytes 2024-09-13T09:58:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1019240448 bytes 2024-09-13T09:58:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1064481792 bytes 2024-09-13T09:58:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1067722752 bytes 2024-09-13T09:58:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1077112832 bytes 2024-09-13T09:58:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1133792256 bytes 2024-09-13T09:58:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1169384448 bytes 2024-09-13T09:58:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1265581056 bytes 2024-09-13T09:58:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1325177856 bytes 2024-09-13T09:58:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1348073472 bytes 2024-09-13T09:58:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1395164160 bytes 2024-09-13T09:58:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1435369472 bytes 2024-09-13T09:58:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1454691328 bytes 2024-09-13T09:58:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1491021824 bytes 2024-09-13T09:58:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1517216768 bytes 2024-09-13T09:58:26.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1530573824 bytes 2024-09-13T09:58:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1548876800 bytes 2024-09-13T09:58:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1572538368 bytes 2024-09-13T09:58:27.081 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1602946048 bytes 2024-09-13T09:58:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1644282880 bytes 2024-09-13T09:58:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1716766720 bytes 2024-09-13T09:58:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1727338496 bytes 2024-09-13T09:58:27.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1820473344 bytes 2024-09-13T09:58:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1848136704 bytes 2024-09-13T09:58:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1896347648 bytes 2024-09-13T09:58:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1952562176 bytes 2024-09-13T09:58:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1965961216 bytes 2024-09-13T09:58:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1989195776 bytes 2024-09-13T09:58:28.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2003957760 bytes 2024-09-13T09:58:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2015666176 bytes 2024-09-13T09:58:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2026795008 bytes 2024-09-13T09:58:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2053090304 bytes 2024-09-13T09:58:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2108780544 bytes 2024-09-13T09:58:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2131282944 bytes 2024-09-13T09:58:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2155173888 bytes 2024-09-13T09:58:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2200535040 bytes 2024-09-13T09:58:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2242239488 bytes 2024-09-13T09:58:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2281273344 bytes 2024-09-13T09:58:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2322648064 bytes 2024-09-13T09:58:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2331971584 bytes 2024-09-13T09:58:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2362661888 bytes 2024-09-13T09:58:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2389267456 bytes 2024-09-13T09:58:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2417720320 bytes 2024-09-13T09:58:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2484239360 bytes 2024-09-13T09:58:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2498531328 bytes 2024-09-13T09:58:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2561797120 bytes 2024-09-13T09:58:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2579685376 bytes 2024-09-13T09:58:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2585759744 bytes 2024-09-13T09:58:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2635081728 bytes 2024-09-13T09:58:30.588 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 2669391872 bytes 2024-09-13T09:58:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 2775609344 bytes 2024-09-13T09:58:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 2816704512 bytes 2024-09-13T09:58:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 2898265088 bytes 2024-09-13T09:58:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 2933295104 bytes 2024-09-13T09:58:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 2963049472 bytes 2024-09-13T09:58:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 2963654656 bytes 2024-09-13T09:58:31.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3019893760 bytes 2024-09-13T09:58:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3045297152 bytes 2024-09-13T09:58:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3053865984 bytes 2024-09-13T09:58:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3054496768 bytes 2024-09-13T09:58:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8200 IOs, 3110493184 bytes 2024-09-13T09:58:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8300 IOs, 3214249984 bytes 2024-09-13T09:58:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8400 IOs, 3232734208 bytes 2024-09-13T09:58:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8500 IOs, 3277645824 bytes 2024-09-13T09:58:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8600 IOs, 3314718720 bytes 2024-09-13T09:58:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8700 IOs, 3385652224 bytes 2024-09-13T09:58:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8800 IOs, 3483286528 bytes 2024-09-13T09:58:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8900 IOs, 3548197888 bytes 2024-09-13T09:58:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9000 IOs, 3578151936 bytes 2024-09-13T09:58:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9100 IOs, 3601465344 bytes 2024-09-13T09:58:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9200 IOs, 3661283328 bytes 2024-09-13T09:58:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9300 IOs, 3702705152 bytes 2024-09-13T09:58:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9400 IOs, 3729801216 bytes 2024-09-13T09:58:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9500 IOs, 3765293056 bytes 2024-09-13T09:58:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9600 IOs, 3825644544 bytes 2024-09-13T09:58:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9700 IOs, 3855203328 bytes 2024-09-13T09:58:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9800 IOs, 3855798272 bytes 2024-09-13T09:58:34.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:9900 IOs, 3909555200 bytes 2024-09-13T09:58:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10000 IOs, 3920920576 bytes 2024-09-13T09:58:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:10100 IOs, 4003871744 bytes 2024-09-13T09:58:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:58:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:58:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:58:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:58:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-09-13T09:58:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:58:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:58:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:58:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=present 2024-09-13T09:58:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:58:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:58:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:58:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_6 2024-09-13T09:58:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:58:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:58:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_6 2024-09-13T09:58:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-13T09:58:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:58:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=128219b443ab 2024-09-13T09:58:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:58:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:58:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_6 present 128219b443ab 2024-09-13T09:58:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:58:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:58:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:58:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=present 2024-09-13T09:58:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=128219b443ab 2024-09-13T09:58:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:58:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:58:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_6 2024-09-13T09:58:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:58:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:58:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_6 2024-09-13T09:58:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-13T09:58:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:58:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=128219b443ab 2024-09-13T09:58:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 128219b443ab 2024-09-13T09:58:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 128219b443ab -o 128219b443ab = 128219b443ab 2024-09-13T09:58:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2024-09-13T09:58:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test present = present 2024-09-13T09:58:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:58:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ disable_mirror cluster2 mirror image_6 2024-09-13T09:58:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:58:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:58:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:58:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-09-13T09:58:35.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T09:58:35.278+0000 7f71725bc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f30196b800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=378 2024-09-13T09:58:35.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T09:58:35.278+0000 7f71725bc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f30196b800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=379, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b8d40c1b-48a7-4e9b-8cbe-df65c34733d8, primary_snap_id=17a, last_copied_object_number=128, snap_seqs={378=18446744073709551614}] 2024-09-13T09:58:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring disabled 2024-09-13T09:58:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-13T09:58:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:58:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:58:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:58:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:58:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:58:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:58:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:58:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_6 2024-09-13T09:58:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:58:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:58:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_6 2024-09-13T09:58:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-13T09:58:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:58:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-13T09:58:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:58:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:58:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:58:40.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-13T09:58:40.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:58:40.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:58:40.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:58:40.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:58:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:58:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:58:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:58:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_6 2024-09-13T09:58:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:58:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:58:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_6 2024-09-13T09:58:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-13T09:58:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:58:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-13T09:58:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:58:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:58:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:58:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:58:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ purge_snapshots cluster2 mirror image_6 2024-09-13T09:58:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:58:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:58:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:58:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-09-13T09:58:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-13T09:58:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:58:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:58:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:58:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:58:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:58:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_6 2024-09-13T09:58:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:58:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:58:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_6 2024-09-13T09:58:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-13T09:58:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-13T09:59:01.677 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5460282644620694) > weight (0.25) 2024-09-13T09:59:01.677 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3547867639932023) > weight (0.25) 2024-09-13T09:59:01.677 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7839113988210071) > weight (0.25) 2024-09-13T09:59:01.678 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.2672456574669254) > weight (0.25) 2024-09-13T09:59:01.678 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2024-09-13T09:59:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-13T09:59:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:59:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:59:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_7 2024-09-13T09:59:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_7 512M 2024-09-13T09:59:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:59:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:59:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:59:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:59:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:59:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:59:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 512M ']' 2024-09-13T09:59:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2024-09-13T09:59:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:59:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_7 2024-09-13T09:59:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_7 2024-09-13T09:59:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:59:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:59:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:59:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-13T09:59:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:59:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-13T09:59:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_7': 2024-09-13T09:59:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T09:59:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:59:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:59:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 199a3cca6d9d 2024-09-13T09:59:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.199a3cca6d9d 2024-09-13T09:59:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:59:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:59:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:59:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:59:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:59:03 2024 2024-09-13T09:59:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:59:03 2024 2024-09-13T09:59:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:59:03 2024 2024-09-13T09:59:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:59:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:59:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 87018ef9-de6c-4a73-adbd-3ffe80db283f 2024-09-13T09:59:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:59:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-13T09:59:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:59:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:59:04.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:59:04.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=6 2024-09-13T09:59:04.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:59:04.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:59:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 19120128 bytes 2024-09-13T09:59:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 84953088 bytes 2024-09-13T09:59:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 106913792 bytes 2024-09-13T09:59:04.798 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 137413632 bytes 2024-09-13T09:59:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 217497600 bytes 2024-09-13T09:59:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 233758720 bytes 2024-09-13T09:59:05.143 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 257221632 bytes 2024-09-13T09:59:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 261122048 bytes 2024-09-13T09:59:05.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 271709184 bytes 2024-09-13T09:59:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 289386496 bytes 2024-09-13T09:59:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 303035392 bytes 2024-09-13T09:59:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 355136512 bytes 2024-09-13T09:59:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 400313344 bytes 2024-09-13T09:59:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 430887936 bytes 2024-09-13T09:59:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 440465408 bytes 2024-09-13T09:59:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 499215360 bytes 2024-09-13T09:59:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 519449600 bytes 2024-09-13T09:59:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 547088384 bytes 2024-09-13T09:59:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 547709952 bytes 2024-09-13T09:59:06.471 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 564486144 bytes 2024-09-13T09:59:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 585381888 bytes 2024-09-13T09:59:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 620657664 bytes 2024-09-13T09:59:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 635432960 bytes 2024-09-13T09:59:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 681659392 bytes 2024-09-13T09:59:06.996 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 684120064 bytes 2024-09-13T09:59:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 751371264 bytes 2024-09-13T09:59:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 831840256 bytes 2024-09-13T09:59:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 862229504 bytes 2024-09-13T09:59:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 900579328 bytes 2024-09-13T09:59:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 901184512 bytes 2024-09-13T09:59:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 955608064 bytes 2024-09-13T09:59:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1001428992 bytes 2024-09-13T09:59:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1054978048 bytes 2024-09-13T09:59:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1089074176 bytes 2024-09-13T09:59:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1118324736 bytes 2024-09-13T09:59:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1129918464 bytes 2024-09-13T09:59:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1189535744 bytes 2024-09-13T09:59:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1236962304 bytes 2024-09-13T09:59:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1252601856 bytes 2024-09-13T09:59:08.918 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1287401472 bytes 2024-09-13T09:59:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1321789440 bytes 2024-09-13T09:59:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1351846912 bytes 2024-09-13T09:59:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1382668288 bytes 2024-09-13T09:59:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1400190976 bytes 2024-09-13T09:59:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1413124096 bytes 2024-09-13T09:59:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1486583808 bytes 2024-09-13T09:59:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1531107328 bytes 2024-09-13T09:59:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 1543312384 bytes 2024-09-13T09:59:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 1614093312 bytes 2024-09-13T09:59:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 1637856256 bytes 2024-09-13T09:59:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 1639094272 bytes 2024-09-13T09:59:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 1679270912 bytes 2024-09-13T09:59:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:59:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:59:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:59:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:59:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-09-13T09:59:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:59:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:59:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=present 2024-09-13T09:59:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:59:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:59:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:59:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_7 2024-09-13T09:59:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:59:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:59:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_7 2024-09-13T09:59:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-13T09:59:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:59:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=128224944359 2024-09-13T09:59:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:59:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:59:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_7 present 128224944359 2024-09-13T09:59:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:59:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:59:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=present 2024-09-13T09:59:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=128224944359 2024-09-13T09:59:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:59:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:59:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_7 2024-09-13T09:59:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:59:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:59:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_7 2024-09-13T09:59:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-13T09:59:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:59:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=128224944359 2024-09-13T09:59:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 128224944359 2024-09-13T09:59:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 128224944359 -o 128224944359 = 128224944359 2024-09-13T09:59:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2024-09-13T09:59:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test present = present 2024-09-13T09:59:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:59:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ disable_mirror cluster2 mirror image_7 2024-09-13T09:59:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:59:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:59:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-09-13T09:59:11.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T09:59:11.262+0000 7f7171dbb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f303e74400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=380 2024-09-13T09:59:11.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T09:59:11.262+0000 7f7171dbb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f303e74400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=381, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b8d40c1b-48a7-4e9b-8cbe-df65c34733d8, primary_snap_id=17c, last_copied_object_number=128, snap_seqs={380=18446744073709551614}] 2024-09-13T09:59:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring disabled 2024-09-13T09:59:16.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-13T09:59:16.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:59:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:59:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:59:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:59:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:59:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:59:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_7 2024-09-13T09:59:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:59:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:59:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_7 2024-09-13T09:59:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-13T09:59:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:59:16.307 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-13T09:59:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:59:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:59:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:59:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-13T09:59:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:59:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:59:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:59:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:59:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:59:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:59:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_7 2024-09-13T09:59:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:59:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:59:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_7 2024-09-13T09:59:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-13T09:59:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:59:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-13T09:59:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:59:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:59:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:59:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:59:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ purge_snapshots cluster2 mirror image_7 2024-09-13T09:59:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:59:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:59:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-09-13T09:59:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-13T09:59:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:59:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:59:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:59:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:59:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_7 2024-09-13T09:59:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:59:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_7 2024-09-13T09:59:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-13T09:59:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-13T09:59:35.476 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-13T09:59:35.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-13T09:59:35.477 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-13T09:59:35.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T09:59:35.476+0000 7f648ebf3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 69716) UID: 1000 2024-09-13T09:59:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-13T09:59:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:59:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T09:59:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_8 2024-09-13T09:59:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_8 512M 2024-09-13T09:59:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:59:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:59:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:59:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:59:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:59:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T09:59:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 512M ']' 2024-09-13T09:59:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2024-09-13T09:59:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T09:59:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_8 2024-09-13T09:59:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_8 2024-09-13T09:59:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:59:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:59:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T09:59:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-13T09:59:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T09:59:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-13T09:59:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_8': 2024-09-13T09:59:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T09:59:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T09:59:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T09:59:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 19ae9d73e50e 2024-09-13T09:59:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.19ae9d73e50e 2024-09-13T09:59:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T09:59:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T09:59:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T09:59:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T09:59:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 09:59:40 2024 2024-09-13T09:59:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 09:59:40 2024 2024-09-13T09:59:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 09:59:40 2024 2024-09-13T09:59:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T09:59:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T09:59:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 02b8c2b3-c09b-417b-8a08-32e5904b6ea9 2024-09-13T09:59:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T09:59:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-13T09:59:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:59:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:59:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T09:59:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=11 2024-09-13T09:59:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T09:59:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-13T09:59:41.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 62817280 bytes 2024-09-13T09:59:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 76904448 bytes 2024-09-13T09:59:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 99124224 bytes 2024-09-13T09:59:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 129794048 bytes 2024-09-13T09:59:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 136317952 bytes 2024-09-13T09:59:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 138717184 bytes 2024-09-13T09:59:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 182488064 bytes 2024-09-13T09:59:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 212869120 bytes 2024-09-13T09:59:42.477 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-13T09:59:42.478 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T09:59:42.478 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-13T09:59:42.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-13T09:59:42.478 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-13T09:59:42.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-13T09:59:42.482 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2024-09-13T09:59:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 276024320 bytes 2024-09-13T09:59:42.709 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 323602432 bytes 2024-09-13T09:59:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 336538624 bytes 2024-09-13T09:59:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 387769344 bytes 2024-09-13T09:59:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 416232448 bytes 2024-09-13T09:59:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 504263680 bytes 2024-09-13T09:59:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 535848960 bytes 2024-09-13T09:59:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 599618560 bytes 2024-09-13T09:59:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 671733760 bytes 2024-09-13T09:59:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 735596544 bytes 2024-09-13T09:59:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 757227520 bytes 2024-09-13T09:59:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 777333760 bytes 2024-09-13T09:59:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 807845888 bytes 2024-09-13T09:59:44.397 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 868217856 bytes 2024-09-13T09:59:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 902888448 bytes 2024-09-13T09:59:44.597 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 905585664 bytes 2024-09-13T09:59:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 951566336 bytes 2024-09-13T09:59:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 978651136 bytes 2024-09-13T09:59:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1056952320 bytes 2024-09-13T09:59:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1078641664 bytes 2024-09-13T09:59:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1104466944 bytes 2024-09-13T09:59:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1113759744 bytes 2024-09-13T09:59:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1192858624 bytes 2024-09-13T09:59:45.584 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1256495104 bytes 2024-09-13T09:59:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1314519040 bytes 2024-09-13T09:59:45.906 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1393810432 bytes 2024-09-13T09:59:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1448428544 bytes 2024-09-13T09:59:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1486062592 bytes 2024-09-13T09:59:46.259 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1498261504 bytes 2024-09-13T09:59:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1518780416 bytes 2024-09-13T09:59:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1533020160 bytes 2024-09-13T09:59:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1598620672 bytes 2024-09-13T09:59:46.773 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1673675776 bytes 2024-09-13T09:59:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1716433920 bytes 2024-09-13T09:59:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1753991168 bytes 2024-09-13T09:59:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1837467648 bytes 2024-09-13T09:59:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 1867770880 bytes 2024-09-13T09:59:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 1896098816 bytes 2024-09-13T09:59:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 1991424000 bytes 2024-09-13T09:59:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2086088704 bytes 2024-09-13T09:59:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2161309696 bytes 2024-09-13T09:59:48.140 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2229133312 bytes 2024-09-13T09:59:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2290877440 bytes 2024-09-13T09:59:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2301717504 bytes 2024-09-13T09:59:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2321487872 bytes 2024-09-13T09:59:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2380385280 bytes 2024-09-13T09:59:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2405498880 bytes 2024-09-13T09:59:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2425669632 bytes 2024-09-13T09:59:48.993 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2462577664 bytes 2024-09-13T09:59:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2517388288 bytes 2024-09-13T09:59:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2565049344 bytes 2024-09-13T09:59:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2594011136 bytes 2024-09-13T09:59:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2604130304 bytes 2024-09-13T09:59:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2626757632 bytes 2024-09-13T09:59:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2685595648 bytes 2024-09-13T09:59:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2760174592 bytes 2024-09-13T09:59:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2760802304 bytes 2024-09-13T09:59:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 2790619136 bytes 2024-09-13T09:59:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 2858767360 bytes 2024-09-13T09:59:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 2905849856 bytes 2024-09-13T09:59:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 2971173888 bytes 2024-09-13T09:59:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 2971764736 bytes 2024-09-13T09:59:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 3021721600 bytes 2024-09-13T09:59:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 3066159104 bytes 2024-09-13T09:59:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 3094716416 bytes 2024-09-13T09:59:51.267 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3200555008 bytes 2024-09-13T09:59:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3216720896 bytes 2024-09-13T09:59:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3317923840 bytes 2024-09-13T09:59:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3344779264 bytes 2024-09-13T09:59:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3408425984 bytes 2024-09-13T09:59:51.988 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3484982272 bytes 2024-09-13T09:59:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3507936256 bytes 2024-09-13T09:59:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3550451712 bytes 2024-09-13T09:59:52.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T09:59:52.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T09:59:52.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T09:59:52.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:59:52.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-09-13T09:59:52.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:59:52.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:52.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:59:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=present 2024-09-13T09:59:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:59:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:59:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:59:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_8 2024-09-13T09:59:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:59:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:59:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_8 2024-09-13T09:59:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-13T09:59:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:59:52.404 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=12826113dc77 2024-09-13T09:59:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:59:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:59:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_8 present 12826113dc77 2024-09-13T09:59:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:59:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:59:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=present 2024-09-13T09:59:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=12826113dc77 2024-09-13T09:59:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:59:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:59:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_8 2024-09-13T09:59:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:59:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:59:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_8 2024-09-13T09:59:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-13T09:59:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:59:52.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=12826113dc77 2024-09-13T09:59:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 12826113dc77 2024-09-13T09:59:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 12826113dc77 -o 12826113dc77 = 12826113dc77 2024-09-13T09:59:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2024-09-13T09:59:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test present = present 2024-09-13T09:59:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:59:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ disable_mirror cluster2 mirror image_8 2024-09-13T09:59:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:59:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:59:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-09-13T09:59:53.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T09:59:53.323+0000 7f71725bc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f301af3c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=382 2024-09-13T09:59:53.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T09:59:53.323+0000 7f71725bc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f301af3c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=383, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b8d40c1b-48a7-4e9b-8cbe-df65c34733d8, primary_snap_id=17e, last_copied_object_number=128, snap_seqs={382=18446744073709551614}] 2024-09-13T09:59:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring disabled 2024-09-13T09:59:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-13T09:59:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:59:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:59:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T09:59:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:59:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T09:59:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:59:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_8 2024-09-13T09:59:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:59:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:59:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_8 2024-09-13T09:59:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-13T09:59:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:59:58.371 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-13T09:59:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T09:59:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T09:59:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T09:59:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-13T09:59:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T09:59:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:59:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T09:59:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T09:59:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T09:59:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T09:59:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_8 2024-09-13T09:59:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T09:59:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T09:59:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_8 2024-09-13T09:59:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-13T09:59:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T09:59:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-13T09:59:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T09:59:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T09:59:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T09:59:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T09:59:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ purge_snapshots cluster2 mirror image_8 2024-09-13T09:59:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:59:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:59:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2024-09-13T09:59:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-13T09:59:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:59:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:59:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T09:59:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T09:59:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_8 2024-09-13T09:59:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T09:59:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T09:59:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_8 2024-09-13T09:59:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-13T09:59:58.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-13T10:00:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-13T10:00:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T10:00:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T10:00:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_9 2024-09-13T10:00:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_9 512M 2024-09-13T10:00:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T10:00:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T10:00:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:00:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T10:00:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T10:00:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T10:00:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T10:00:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 512M ']' 2024-09-13T10:00:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2024-09-13T10:00:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T10:00:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_9 2024-09-13T10:00:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_9 2024-09-13T10:00:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T10:00:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:00:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T10:00:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T10:00:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-13T10:00:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T10:00:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-13T10:00:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_9': 2024-09-13T10:00:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T10:00:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T10:00:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T10:00:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 19c83a6cb158 2024-09-13T10:00:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.19c83a6cb158 2024-09-13T10:00:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T10:00:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T10:00:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T10:00:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T10:00:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 10:00:22 2024 2024-09-13T10:00:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 10:00:22 2024 2024-09-13T10:00:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 10:00:22 2024 2024-09-13T10:00:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T10:00:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T10:00:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: 1f187a3b-c825-4f36-8a53-bc4d6fa2538b 2024-09-13T10:00:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T10:00:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-13T10:00:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T10:00:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:00:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T10:00:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T10:00:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=6 2024-09-13T10:00:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T10:00:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-13T10:00:23.613 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 72626176 bytes 2024-09-13T10:00:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 148246528 bytes 2024-09-13T10:00:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 148843520 bytes 2024-09-13T10:00:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 171575296 bytes 2024-09-13T10:00:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 283840512 bytes 2024-09-13T10:00:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 371347456 bytes 2024-09-13T10:00:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 438455296 bytes 2024-09-13T10:00:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 494257152 bytes 2024-09-13T10:00:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 505337856 bytes 2024-09-13T10:00:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 535782400 bytes 2024-09-13T10:00:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 635676672 bytes 2024-09-13T10:00:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 678408192 bytes 2024-09-13T10:00:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 703509504 bytes 2024-09-13T10:00:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 768878592 bytes 2024-09-13T10:00:25.374 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 769506304 bytes 2024-09-13T10:00:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 845001728 bytes 2024-09-13T10:00:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 845614080 bytes 2024-09-13T10:00:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 874118144 bytes 2024-09-13T10:00:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 928703488 bytes 2024-09-13T10:00:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 959855616 bytes 2024-09-13T10:00:26.087 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 994073600 bytes 2024-09-13T10:00:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 1057331200 bytes 2024-09-13T10:00:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 1155729408 bytes 2024-09-13T10:00:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 1222042624 bytes 2024-09-13T10:00:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1263075328 bytes 2024-09-13T10:00:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1313576960 bytes 2024-09-13T10:00:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1351729152 bytes 2024-09-13T10:00:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1369364480 bytes 2024-09-13T10:00:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1394022400 bytes 2024-09-13T10:00:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1439224832 bytes 2024-09-13T10:00:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1454283776 bytes 2024-09-13T10:00:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1506354176 bytes 2024-09-13T10:00:27.604 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1539393536 bytes 2024-09-13T10:00:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1588804608 bytes 2024-09-13T10:00:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1653927936 bytes 2024-09-13T10:00:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1677427712 bytes 2024-09-13T10:00:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1739651072 bytes 2024-09-13T10:00:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1821985792 bytes 2024-09-13T10:00:28.453 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1868900352 bytes 2024-09-13T10:00:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1904016384 bytes 2024-09-13T10:00:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1908676608 bytes 2024-09-13T10:00:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1965661184 bytes 2024-09-13T10:00:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1993292800 bytes 2024-09-13T10:00:29.002 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 2014637056 bytes 2024-09-13T10:00:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2074274816 bytes 2024-09-13T10:00:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2116865024 bytes 2024-09-13T10:00:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2154695680 bytes 2024-09-13T10:00:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T10:00:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T10:00:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T10:00:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T10:00:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2024-09-13T10:00:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T10:00:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:00:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T10:00:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=present 2024-09-13T10:00:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T10:00:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T10:00:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T10:00:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_9 2024-09-13T10:00:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T10:00:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T10:00:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_9 2024-09-13T10:00:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-13T10:00:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T10:00:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=1282cf05338c 2024-09-13T10:00:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T10:00:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T10:00:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_9 present 1282cf05338c 2024-09-13T10:00:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T10:00:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:00:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T10:00:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=present 2024-09-13T10:00:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=1282cf05338c 2024-09-13T10:00:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T10:00:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T10:00:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_9 2024-09-13T10:00:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T10:00:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T10:00:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_9 2024-09-13T10:00:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-13T10:00:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T10:00:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=1282cf05338c 2024-09-13T10:00:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1282cf05338c 2024-09-13T10:00:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 1282cf05338c -o 1282cf05338c = 1282cf05338c 2024-09-13T10:00:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2024-09-13T10:00:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test present = present 2024-09-13T10:00:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T10:00:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ disable_mirror cluster2 mirror image_9 2024-09-13T10:00:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T10:00:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:00:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T10:00:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2024-09-13T10:00:29.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:00:29.828+0000 7f7171dbb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f301675400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=384 2024-09-13T10:00:29.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:00:29.828+0000 7f7171dbb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f301675400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=385, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b8d40c1b-48a7-4e9b-8cbe-df65c34733d8, primary_snap_id=180, last_copied_object_number=128, snap_seqs={384=18446744073709551614}] 2024-09-13T10:00:31.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:00:31.750+0000 7fb782542640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f265002340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T10:00:31.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:00:31.750+0000 7fb788d4f640 -1 rbd::mirror::ImageReplayer: 0x55f264552f00 [3/1f187a3b-c825-4f36-8a53-bc4d6fa2538b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T10:00:32.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:00:32.551+0000 7fb782d43640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f264f4d520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-13T10:00:32.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:00:32.551+0000 7fb789550640 -1 rbd::mirror::ImageReplayer: 0x55f264552f00 [3/1f187a3b-c825-4f36-8a53-bc4d6fa2538b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-13T10:00:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring disabled 2024-09-13T10:00:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-13T10:00:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T10:00:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:00:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T10:00:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T10:00:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T10:00:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T10:00:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T10:00:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_9 2024-09-13T10:00:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T10:00:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T10:00:34.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_9 2024-09-13T10:00:34.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-13T10:00:34.842 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T10:00:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-13T10:00:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T10:00:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T10:00:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T10:00:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-13T10:00:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T10:00:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:00:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T10:00:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T10:00:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T10:00:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T10:00:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T10:00:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_9 2024-09-13T10:00:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T10:00:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T10:00:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_9 2024-09-13T10:00:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-13T10:00:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T10:00:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-13T10:00:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T10:00:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T10:00:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T10:00:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T10:00:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ purge_snapshots cluster2 mirror image_9 2024-09-13T10:00:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T10:00:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:00:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T10:00:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2024-09-13T10:00:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-13T10:00:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T10:00:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:00:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T10:00:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T10:00:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T10:00:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_9 2024-09-13T10:00:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T10:00:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:00:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_9 2024-09-13T10:00:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-13T10:00:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-13T10:00:49.679 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.591298638719103) > weight (0.25) 2024-09-13T10:00:49.679 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6498145867385932) > weight (0.25) 2024-09-13T10:00:49.679 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9680421435723693) > weight (0.25) 2024-09-13T10:00:49.679 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.4701806576266473) > weight (0.25) 2024-09-13T10:00:49.679 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2024-09-13T10:00:59.468 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-13T10:00:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T10:00:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for i in `seq 1 10` 2024-09-13T10:00:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image=image_10 2024-09-13T10:00:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ create_image cluster2 mirror image_10 512M 2024-09-13T10:00:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T10:00:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T10:00:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:00:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T10:00:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T10:00:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T10:00:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local size=128 2024-09-13T10:00:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -n 512M ']' 2024-09-13T10:00:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ size=512M 2024-09-13T10:00:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ shift 2024-09-13T10:00:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_10 2024-09-13T10:00:59.483 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.376486731969193) > weight (0.25) 2024-09-13T10:00:59.484 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8944264661939825) > weight (0.25) 2024-09-13T10:00:59.484 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-13T10:00:59.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-13T10:00:59.485 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-13T10:00:59.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T10:00:59.484+0000 7f894331b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 69277) UID: 1000 2024-09-13T10:00:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ enable_mirror cluster2 mirror image_10 2024-09-13T10:00:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T10:00:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:00:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T10:00:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local mode=snapshot 2024-09-13T10:00:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-13T10:01:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring enabled 2024-09-13T10:01:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-13T10:01:00.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:rbd image 'image_10': 2024-09-13T10:01:00.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: size 512 MiB in 128 objects 2024-09-13T10:01:00.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: order 22 (4 MiB objects) 2024-09-13T10:01:00.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: snapshot_count: 1 2024-09-13T10:01:00.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: id: 19ddef746825 2024-09-13T10:01:00.323 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: block_name_prefix: rbd_data.19ddef746825 2024-09-13T10:01:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: format: 2 2024-09-13T10:01:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: features: layering, exclusive-lock 2024-09-13T10:01:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: op_features: 2024-09-13T10:01:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: flags: 2024-09-13T10:01:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: create_timestamp: Fri Sep 13 10:00:59 2024 2024-09-13T10:01:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: access_timestamp: Fri Sep 13 10:00:59 2024 2024-09-13T10:01:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: modify_timestamp: Fri Sep 13 10:00:59 2024 2024-09-13T10:01:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring state: enabled 2024-09-13T10:01:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring mode: snapshot 2024-09-13T10:01:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring global id: a2022c7c-7cd1-4da5-8d83-99da290367c3 2024-09-13T10:01:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout: mirroring primary: true 2024-09-13T10:01:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-13T10:01:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T10:01:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:01:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T10:01:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-13T10:01:00.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local duration=10 2024-09-13T10:01:00.328 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T10:01:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-13T10:01:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:100 IOs, 40208384 bytes 2024-09-13T10:01:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:200 IOs, 74194944 bytes 2024-09-13T10:01:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:300 IOs, 122237952 bytes 2024-09-13T10:01:00.807 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:400 IOs, 127417344 bytes 2024-09-13T10:01:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:500 IOs, 188615680 bytes 2024-09-13T10:01:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:600 IOs, 223355904 bytes 2024-09-13T10:01:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:700 IOs, 244583424 bytes 2024-09-13T10:01:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:800 IOs, 341122048 bytes 2024-09-13T10:01:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:900 IOs, 396467200 bytes 2024-09-13T10:01:01.566 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1000 IOs, 439682048 bytes 2024-09-13T10:01:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1100 IOs, 452618240 bytes 2024-09-13T10:01:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1200 IOs, 480661504 bytes 2024-09-13T10:01:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1300 IOs, 533342208 bytes 2024-09-13T10:01:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1400 IOs, 565813248 bytes 2024-09-13T10:01:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1500 IOs, 597713920 bytes 2024-09-13T10:01:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1600 IOs, 618502144 bytes 2024-09-13T10:01:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1700 IOs, 691688448 bytes 2024-09-13T10:01:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1800 IOs, 743435264 bytes 2024-09-13T10:01:02.592 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:1900 IOs, 766511104 bytes 2024-09-13T10:01:02.676 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2000 IOs, 767140864 bytes 2024-09-13T10:01:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2100 IOs, 787154944 bytes 2024-09-13T10:01:02.915 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2200 IOs, 840687616 bytes 2024-09-13T10:01:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2300 IOs, 917857280 bytes 2024-09-13T10:01:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2400 IOs, 976015360 bytes 2024-09-13T10:01:03.334 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2500 IOs, 1031550976 bytes 2024-09-13T10:01:03.473 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2600 IOs, 1094982656 bytes 2024-09-13T10:01:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2700 IOs, 1102171136 bytes 2024-09-13T10:01:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2800 IOs, 1121753088 bytes 2024-09-13T10:01:03.795 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:2900 IOs, 1187309568 bytes 2024-09-13T10:01:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3000 IOs, 1194332160 bytes 2024-09-13T10:01:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3100 IOs, 1303032832 bytes 2024-09-13T10:01:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3200 IOs, 1378936832 bytes 2024-09-13T10:01:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3300 IOs, 1471924224 bytes 2024-09-13T10:01:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3400 IOs, 1525822464 bytes 2024-09-13T10:01:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3500 IOs, 1598885888 bytes 2024-09-13T10:01:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3600 IOs, 1613067264 bytes 2024-09-13T10:01:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3700 IOs, 1627631616 bytes 2024-09-13T10:01:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3800 IOs, 1680864256 bytes 2024-09-13T10:01:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:3900 IOs, 1733422080 bytes 2024-09-13T10:01:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4000 IOs, 1804786688 bytes 2024-09-13T10:01:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4100 IOs, 1854648320 bytes 2024-09-13T10:01:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4200 IOs, 1900993536 bytes 2024-09-13T10:01:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4300 IOs, 1976956928 bytes 2024-09-13T10:01:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4400 IOs, 1985341440 bytes 2024-09-13T10:01:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4500 IOs, 2036418560 bytes 2024-09-13T10:01:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4600 IOs, 2091399168 bytes 2024-09-13T10:01:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4700 IOs, 2114659328 bytes 2024-09-13T10:01:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4800 IOs, 2159634432 bytes 2024-09-13T10:01:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:4900 IOs, 2222182400 bytes 2024-09-13T10:01:06.460 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5000 IOs, 2230557696 bytes 2024-09-13T10:01:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5100 IOs, 2310594560 bytes 2024-09-13T10:01:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5200 IOs, 2393017344 bytes 2024-09-13T10:01:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5300 IOs, 2450338816 bytes 2024-09-13T10:01:07.010 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5400 IOs, 2475331584 bytes 2024-09-13T10:01:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5500 IOs, 2544602112 bytes 2024-09-13T10:01:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5600 IOs, 2615421952 bytes 2024-09-13T10:01:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5700 IOs, 2641131520 bytes 2024-09-13T10:01:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5800 IOs, 2702148608 bytes 2024-09-13T10:01:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:5900 IOs, 2733181952 bytes 2024-09-13T10:01:07.730 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2024-09-13T10:01:07.726+0000 7f3610499640 -1 reset not still connected to 0x55cdf8a829c0 2024-09-13T10:01:07.730 INFO:tasks.ceph.cluster2.osd.1.smithi190.stderr:2024-09-13T10:01:07.726+0000 7f3610499640 -1 reset not still connected to 0x55cdfdecf040 2024-09-13T10:01:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6000 IOs, 2777597952 bytes 2024-09-13T10:01:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6100 IOs, 2821352448 bytes 2024-09-13T10:01:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6200 IOs, 2872147968 bytes 2024-09-13T10:01:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6300 IOs, 2891762688 bytes 2024-09-13T10:01:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6400 IOs, 2961870848 bytes 2024-09-13T10:01:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6500 IOs, 2993647616 bytes 2024-09-13T10:01:08.486 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-13T10:01:08.486 INFO:teuthology.orchestra.run:waiting for 600 2024-09-13T10:01:08.486 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-13T10:01:08.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-13T10:01:08.487 DEBUG:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-13T10:01:08.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-13T10:01:08.491 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2024-09-13T10:01:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6600 IOs, 3030447104 bytes 2024-09-13T10:01:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6700 IOs, 3081981952 bytes 2024-09-13T10:01:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6800 IOs, 3167576064 bytes 2024-09-13T10:01:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:6900 IOs, 3217332224 bytes 2024-09-13T10:01:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7000 IOs, 3259474944 bytes 2024-09-13T10:01:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7100 IOs, 3268473856 bytes 2024-09-13T10:01:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7200 IOs, 3286701056 bytes 2024-09-13T10:01:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7300 IOs, 3310217216 bytes 2024-09-13T10:01:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7400 IOs, 3336318976 bytes 2024-09-13T10:01:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7500 IOs, 3390518272 bytes 2024-09-13T10:01:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7600 IOs, 3459916800 bytes 2024-09-13T10:01:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7700 IOs, 3503396864 bytes 2024-09-13T10:01:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7800 IOs, 3569218560 bytes 2024-09-13T10:01:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:7900 IOs, 3593325568 bytes 2024-09-13T10:01:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8000 IOs, 3600842752 bytes 2024-09-13T10:01:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:8100 IOs, 3637513216 bytes 2024-09-13T10:01:10.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=124 2024-09-13T10:01:10.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T10:01:10.336 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 124 -eq 124 ']' 2024-09-13T10:01:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T10:01:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2024-09-13T10:01:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T10:01:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:01:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T10:01:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=present 2024-09-13T10:01:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T10:01:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T10:01:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T10:01:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_10 2024-09-13T10:01:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T10:01:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T10:01:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_10 2024-09-13T10:01:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-13T10:01:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T10:01:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id=1282da6e374f 2024-09-13T10:01:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T10:01:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T10:01:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_10 present 1282da6e374f 2024-09-13T10:01:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T10:01:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:01:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T10:01:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=present 2024-09-13T10:01:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id=1282da6e374f 2024-09-13T10:01:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T10:01:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T10:01:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_10 2024-09-13T10:01:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T10:01:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T10:01:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_10 2024-09-13T10:01:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-13T10:01:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T10:01:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id=1282da6e374f 2024-09-13T10:01:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1282da6e374f 2024-09-13T10:01:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -z 1282da6e374f -o 1282da6e374f = 1282da6e374f 2024-09-13T10:01:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_state=present 2024-09-13T10:01:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test present = present 2024-09-13T10:01:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T10:01:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ disable_mirror cluster2 mirror image_10 2024-09-13T10:01:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T10:01:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:01:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T10:01:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2024-09-13T10:01:11.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:11.294+0000 7f71725bc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f301295c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=386 2024-09-13T10:01:11.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:11.294+0000 7f71725bc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f301295c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=387, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b8d40c1b-48a7-4e9b-8cbe-df65c34733d8, primary_snap_id=182, last_copied_object_number=128, snap_seqs={386=18446744073709551614}] 2024-09-13T10:01:16.297 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:Mirroring disabled 2024-09-13T10:01:16.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-13T10:01:16.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T10:01:16.305 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:01:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T10:01:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local state=deleted 2024-09-13T10:01:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T10:01:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local s 2024-09-13T10:01:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T10:01:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_10 2024-09-13T10:01:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T10:01:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T10:01:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_10 2024-09-13T10:01:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-13T10:01:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T10:01:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-13T10:01:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ image_id= 2024-09-13T10:01:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-13T10:01:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0.1 2024-09-13T10:01:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-13T10:01:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T10:01:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:01:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T10:01:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local test_state=deleted 2024-09-13T10:01:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image_id= 2024-09-13T10:01:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_state=deleted 2024-09-13T10:01:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local current_image_id 2024-09-13T10:01:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ get_image_id cluster1 mirror image_10 2024-09-13T10:01:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T10:01:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T10:01:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local image=image_10 2024-09-13T10:01:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-13T10:01:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-13T10:01:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-13T10:01:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ current_image_id= 2024-09-13T10:01:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n '' 2024-09-13T10:01:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test deleted = deleted 2024-09-13T10:01:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T10:01:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ purge_snapshots cluster2 mirror image_10 2024-09-13T10:01:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T10:01:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:01:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T10:01:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2024-09-13T10:01:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-13T10:01:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T10:01:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:01:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T10:01:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-13T10:01:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T10:01:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ remove_image cluster2 mirror image_10 2024-09-13T10:01:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T10:01:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:01:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local image=image_10 2024-09-13T10:01:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-13T10:01:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-13T10:01:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-13T10:01:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T10:01:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-09-13T10:01:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ tee -a /tmp/tmp.V7YewHEkvl/rbd-mirror.test.log 2024-09-13T10:01:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ date '+%F %T' 2024-09-13T10:01:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo 2024-09-13 10:01:39 TEST: check if removed 'images'\''' OMAP are removed 2024-09-13T10:01:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:2024-09-13 10:01:39 TEST: check if removed images' OMAP are removed 2024-09-13T10:01:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-09-13T10:01:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T10:01:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:01:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-09-13T10:01:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T10:01:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:01:39.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirroring 2024-09-13T10:01:39.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local filter=status_global 2024-09-13T10:01:39.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-13T10:01:39.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T10:01:39.754 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T10:01:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-09-13T10:01:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T10:01:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T10:01:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local obj_name=rbd_mirroring 2024-09-13T10:01:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local filter=status_global 2024-09-13T10:01:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-13T10:01:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T10:01:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T10:01:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local obj_name=rbd_mirroring 2024-09-13T10:01:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-13T10:01:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -c status_global 2024-09-13T10:01:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 = 0 2024-09-13T10:01:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=0 2024-09-13T10:01:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T10:01:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 0 -eq 0 ']' 2024-09-13T10:01:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T10:01:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-09-13T10:01:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T10:01:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:01:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirroring 2024-09-13T10:01:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local filter=image_ 2024-09-13T10:01:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-13T10:01:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T10:01:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T10:01:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-09-13T10:01:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T10:01:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T10:01:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local obj_name=rbd_mirroring 2024-09-13T10:01:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local filter=image_ 2024-09-13T10:01:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-13T10:01:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T10:01:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T10:01:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local obj_name=rbd_mirroring 2024-09-13T10:01:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-13T10:01:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -c image_ 2024-09-13T10:01:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 = 0 2024-09-13T10:01:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=0 2024-09-13T10:01:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T10:01:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 0 -eq 0 ']' 2024-09-13T10:01:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T10:01:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-09-13T10:01:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T10:01:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:01:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirror_leader 2024-09-13T10:01:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local filter=image_map 2024-09-13T10:01:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-13T10:01:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T10:01:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T10:01:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-13T10:01:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T10:01:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T10:01:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local obj_name=rbd_mirror_leader 2024-09-13T10:01:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local filter=image_map 2024-09-13T10:01:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-13T10:01:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster1 2024-09-13T10:01:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T10:01:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local obj_name=rbd_mirror_leader 2024-09-13T10:01:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -c image_map 2024-09-13T10:01:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-13T10:01:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 = 0 2024-09-13T10:01:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=0 2024-09-13T10:01:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T10:01:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 0 -eq 0 ']' 2024-09-13T10:01:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T10:01:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-09-13T10:01:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T10:01:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:01:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-09-13T10:01:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T10:01:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:01:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirroring 2024-09-13T10:01:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local filter=status_global 2024-09-13T10:01:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-13T10:01:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T10:01:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T10:01:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-09-13T10:01:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster2 2024-09-13T10:01:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T10:01:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local obj_name=rbd_mirroring 2024-09-13T10:01:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local filter=status_global 2024-09-13T10:01:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-13T10:01:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster2 2024-09-13T10:01:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T10:01:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local obj_name=rbd_mirroring 2024-09-13T10:01:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-13T10:01:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -c status_global 2024-09-13T10:01:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 = 0 2024-09-13T10:01:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=0 2024-09-13T10:01:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T10:01:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 0 -eq 0 ']' 2024-09-13T10:01:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T10:01:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-09-13T10:01:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T10:01:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:01:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirroring 2024-09-13T10:01:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local filter=image_ 2024-09-13T10:01:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-13T10:01:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T10:01:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T10:01:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-09-13T10:01:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster2 2024-09-13T10:01:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T10:01:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local obj_name=rbd_mirroring 2024-09-13T10:01:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local filter=image_ 2024-09-13T10:01:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-13T10:01:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster2 2024-09-13T10:01:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T10:01:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local obj_name=rbd_mirroring 2024-09-13T10:01:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-13T10:01:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -c image_ 2024-09-13T10:01:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 = 0 2024-09-13T10:01:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=0 2024-09-13T10:01:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T10:01:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 0 -eq 0 ']' 2024-09-13T10:01:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T10:01:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-09-13T10:01:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T10:01:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local pool=mirror 2024-09-13T10:01:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local obj_name=rbd_mirror_leader 2024-09-13T10:01:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local filter=image_map 2024-09-13T10:01:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-13T10:01:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ sleep 0 2024-09-13T10:01:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T10:01:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-09-13T10:01:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster2 2024-09-13T10:01:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T10:01:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local obj_name=rbd_mirror_leader 2024-09-13T10:01:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local filter=image_map 2024-09-13T10:01:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-09-13T10:01:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local cluster=cluster2 2024-09-13T10:01:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local pool=mirror 2024-09-13T10:01:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ local obj_name=rbd_mirror_leader 2024-09-13T10:01:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-09-13T10:01:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ grep -c image_map 2024-09-13T10:01:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test 0 = 0 2024-09-13T10:01:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ error_code=0 2024-09-13T10:01:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set -e 2024-09-13T10:01:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 0 -eq 0 ']' 2024-09-13T10:01:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 0 2024-09-13T10:01:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ cleanup 0 2024-09-13T10:01:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local error_code=0 2024-09-13T10:01:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ set +e 2024-09-13T10:01:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 0 -ne 0 ']' 2024-09-13T10:01:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -z '' ']' 2024-09-13T10:01:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster instance 2024-09-13T10:01:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= 2024-09-13T10:01:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-13T10:01:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:pool 'mirror' does not exist 2024-09-13T10:01:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= 2024-09-13T10:01:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-13T10:01:40.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:40.879+0000 7f717edd5640 -1 rbd::mirror::LeaderWatcher: 0x55f300909b00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-13T10:01:40.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:40.880+0000 7f71785c8640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55f301fa18c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-13T10:01:40.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:40.880+0000 7f71785c8640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55f3048b1680 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-13T10:01:40.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:40.880+0000 7f71785c8640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55f3048b06c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-13T10:01:40.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:40.880+0000 7f71785c8640 -1 rbd::mirror::PoolWatcher: 0x55f301726f00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-13T10:01:40.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:40.880+0000 7f71785c8640 -1 rbd::mirror::PoolWatcher: 0x55f301726d20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-13T10:01:40.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:40.880+0000 7f71785c8640 -1 rbd::mirror::PoolWatcher: 0x55f3014bba40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-13T10:01:40.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:40.884+0000 7f717f5d6640 -1 rbd::mirror::LeaderWatcher: 0x55f300909b00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-13T10:01:40.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:40.884+0000 7f717f5d6640 -1 rbd::mirror::LeaderWatcher: 0x55f300909b00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-13T10:01:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:pool 'mirror' does not exist 2024-09-13T10:01:41.482 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= 2024-09-13T10:01:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-13T10:01:42.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.737+0000 7fdb476cd640 -1 rbd::mirror::RemotePoolPoller: 0x564e636c4340 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-09-13T10:01:42.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.738+0000 7fdb476cd640 -1 rbd::mirror::RemotePoolPoller: 0x564e636c4340 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-09-13T10:01:42.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.738+0000 7fdb476cd640 -1 rbd::mirror::PoolReplayer: 0x564e6455a000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-09-13T10:01:42.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.738+0000 7fdb4d6d9640 -1 rbd::mirror::InstanceWatcher: 0x564e64892380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:42.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.738+0000 7fdb4d6d9640 -1 rbd::mirror::InstanceWatcher: 0x564e64892700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:42.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.739+0000 7fdb46ecc640 -1 rbd::mirror::MirrorStatusUpdater 0x564e648de360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:42.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.739+0000 7fdb46ecc640 -1 rbd::mirror::MirrorStatusUpdater 0x564e648de5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:42.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.739+0000 7fdb546e7640 -1 rbd::mirror::NamespaceReplayer: 0x564e648889c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:42.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.739+0000 7fdb546e7640 -1 rbd::mirror::NamespaceReplayer: 0x564e64888b60 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:42.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.739+0000 7fdb4d6d9640 -1 rbd::mirror::MirrorStatusUpdater 0x564e648de000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:42.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.739+0000 7fdb53ee6640 -1 rbd::mirror::NamespaceReplayer: 0x564e648889c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:42.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.827+0000 7fdb39eb2640 -1 rbd::mirror::RemotePoolPoller: 0x564e636c5930 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-09-13T10:01:42.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.827+0000 7fdb39eb2640 -1 rbd::mirror::RemotePoolPoller: 0x564e636c5930 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-09-13T10:01:42.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.827+0000 7fdb39eb2640 -1 rbd::mirror::PoolReplayer: 0x564e648a4000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-09-13T10:01:42.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.850+0000 7fdb53ee6640 -1 rbd::mirror::LeaderWatcher: 0x564e64661b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-13T10:01:42.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.850+0000 7fdb4d6d9640 -1 rbd::mirror::InstanceWatcher: 0x564e64892000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:42.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.851+0000 7fdb46ecc640 -1 rbd::mirror::MirrorStatusUpdater 0x564e6464c6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:42.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.851+0000 7fdb546e7640 -1 rbd::mirror::NamespaceReplayer: 0x564e64888000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:42.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.851+0000 7fdb4deda640 -1 rbd::mirror::MirrorStatusUpdater 0x564e645ca7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:42.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.851+0000 7fdb53ee6640 -1 rbd::mirror::NamespaceReplayer: 0x564e64888000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:42.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.853+0000 7fdb57f44640 -1 rbd::mirror::Mirror: 0x564e63743c20 update_pool_replayers: restarting failed pool replayer for uuid: 25e58861-347a-4a47-a9c4-e461fae0678b cluster: cluster1 client: client.mirror 2024-09-13T10:01:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:pool 'mirror_parent' does not exist 2024-09-13T10:01:42.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:42.886+0000 7fdb57f44640 -1 rbd::mirror::PoolReplayer: 0x564e648a4000 init: error accessing remote pool mirror_parent: (2) No such file or directory 2024-09-13T10:01:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ CEPH_ARGS= 2024-09-13T10:01:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-13T10:01:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:pool 'mirror_parent' does not exist 2024-09-13T10:01:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-13T10:01:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stop_mirrors cluster1 2024-09-13T10:01:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1 2024-09-13T10:01:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local sig= 2024-09-13T10:01:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ seq 0 3 2024-09-13T10:01:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T10:01:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stop_mirror cluster1:0 '' 2024-09-13T10:01:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:0 2024-09-13T10:01:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local sig= 2024-09-13T10:01:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2024-09-13T10:01:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2024-09-13T10:01:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T10:01:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stop_mirror cluster1:1 '' 2024-09-13T10:01:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:1 2024-09-13T10:01:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local sig= 2024-09-13T10:01:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2024-09-13T10:01:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2024-09-13T10:01:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T10:01:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stop_mirror cluster1:2 '' 2024-09-13T10:01:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:2 2024-09-13T10:01:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local sig= 2024-09-13T10:01:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2024-09-13T10:01:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2024-09-13T10:01:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T10:01:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stop_mirror cluster1:3 '' 2024-09-13T10:01:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster1:3 2024-09-13T10:01:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local sig= 2024-09-13T10:01:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2024-09-13T10:01:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2024-09-13T10:01:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-13T10:01:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stop_mirrors cluster2 2024-09-13T10:01:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2 2024-09-13T10:01:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local sig= 2024-09-13T10:01:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:++ seq 0 3 2024-09-13T10:01:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T10:01:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stop_mirror cluster2:0 '' 2024-09-13T10:01:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2:0 2024-09-13T10:01:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local sig= 2024-09-13T10:01:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2024-09-13T10:01:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2024-09-13T10:01:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T10:01:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stop_mirror cluster2:1 '' 2024-09-13T10:01:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2:1 2024-09-13T10:01:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local sig= 2024-09-13T10:01:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2024-09-13T10:01:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2024-09-13T10:01:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T10:01:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stop_mirror cluster2:2 '' 2024-09-13T10:01:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2:2 2024-09-13T10:01:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local sig= 2024-09-13T10:01:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2024-09-13T10:01:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2024-09-13T10:01:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-13T10:01:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ stop_mirror cluster2:3 '' 2024-09-13T10:01:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local cluster=cluster2:3 2024-09-13T10:01:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ local sig= 2024-09-13T10:01:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test -n 1 2024-09-13T10:01:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ return 2024-09-13T10:01:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' -z 1 ']' 2024-09-13T10:01:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ test '' = /tmp/tmp.V7YewHEkvl 2024-09-13T10:01:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ rm -Rf /tmp/tmp.V7YewHEkvl 2024-09-13T10:01:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stdout:OK 2024-09-13T10:01:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ '[' 0 -eq 0 ']' 2024-09-13T10:01:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ echo OK 2024-09-13T10:01:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi190.stderr:+ exit 0 2024-09-13T10:01:43.507 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-13T10:01:43.507 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-09-13T10:01:43.820 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-09-13T10:01:43.820 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-09-13T10:01:44.061 DEBUG:teuthology.parallel:result is None 2024-09-13T10:01:44.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-13T10:01:44.087 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-13T10:01:44.088 DEBUG:teuthology.orchestra.run.smithi190:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-13T10:01:44.148 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-13T10:01:44.148 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-13T10:01:44.159 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-13T10:01:44.159 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 18 2024-09-13T10:01:44.159 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-13T10:01:44.160 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-13T10:01:44.198 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-13T10:01:44.198 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 16 2024-09-13T10:01:44.199 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-13T10:01:44.199 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-13T10:01:44.269 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2024-09-13T10:01:44.269 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T10:01:44.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.208+0000 7f663f8c5640 -1 rbd::mirror::RemotePoolPoller: 0x5599d21781a0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-09-13T10:01:44.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.208+0000 7f66400c6640 -1 rbd::mirror::RemotePoolPoller: 0x5599d21781a0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-09-13T10:01:44.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.208+0000 7f66400c6640 -1 rbd::mirror::PoolReplayer: 0x5599d2fe0000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-09-13T10:01:44.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.209+0000 7f664d0e0640 -1 rbd::mirror::InstanceWatcher: 0x5599d3322380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6068 object: (2) No such file or directory 2024-09-13T10:01:44.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.209+0000 7f664d0e0640 -1 rbd::mirror::InstanceWatcher: 0x5599d3812000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6068 object: (2) No such file or directory 2024-09-13T10:01:44.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.209+0000 7f66468d3640 -1 rbd::mirror::InstanceWatcher: 0x5599d3322380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.209+0000 7f66468d3640 -1 rbd::mirror::InstanceWatcher: 0x5599d3812000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.209+0000 7f66460d2640 -1 rbd::mirror::MirrorStatusUpdater 0x5599d3390240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.209+0000 7f66460d2640 -1 rbd::mirror::MirrorStatusUpdater 0x5599d3390360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.209+0000 7f664d0e0640 -1 rbd::mirror::NamespaceReplayer: 0x5599d3316820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.209+0000 7f664d0e0640 -1 rbd::mirror::NamespaceReplayer: 0x5599d33169c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.269+0000 7fb8e4b90640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 58195) UID: 1000 2024-09-13T10:01:44.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.269+0000 7fb8ddb82640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55594d156000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-13T10:01:44.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.269+0000 7fb8ddb82640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55594d0b6d80 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-13T10:01:44.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.270+0000 7fb8dd381640 -1 rbd::mirror::PoolWatcher: 0x55594bf7e3c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-13T10:01:44.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.270+0000 7fb8dd381640 -1 rbd::mirror::PoolWatcher: 0x55594d7a4b40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-13T10:01:44.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.270+0000 7fb8e438f640 -1 rbd::mirror::InstanceWatcher: 0x55594d0e2a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5724 object: (2) No such file or directory 2024-09-13T10:01:44.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.270+0000 7fb8d7375640 -1 rbd::mirror::PoolWatcher: 0x55594d71ed20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-13T10:01:44.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.270+0000 7fb8d7375640 -1 rbd::mirror::PoolWatcher: 0x55594d7a4000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-13T10:01:44.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.270+0000 7fb8d0367640 -1 rbd::mirror::InstanceWatcher: 0x55594d0e2a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.270+0000 7fb8e3b8e640 -1 rbd::mirror::InstanceWatcher: 0x55594d0e2380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5722 object: (2) No such file or directory 2024-09-13T10:01:44.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.270+0000 7fb8e3b8e640 -1 rbd::mirror::InstanceWatcher: 0x55594d0e2700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5722 object: (2) No such file or directory 2024-09-13T10:01:44.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.270+0000 7fb8c9b5a640 -1 rbd::mirror::MirrorStatusUpdater 0x55594d33e7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.270+0000 7fb8e438f640 -1 rbd::mirror::NamespaceReplayer: 0x55594d4ee4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.271+0000 7fb8ddb82640 -1 rbd::mirror::InstanceWatcher: 0x55594d0e2380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.271+0000 7fb8ddb82640 -1 rbd::mirror::InstanceWatcher: 0x55594d0e2700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.271+0000 7fb8d6b74640 -1 rbd::mirror::MirrorStatusUpdater 0x55594d1325a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.271+0000 7fb8d6b74640 -1 rbd::mirror::MirrorStatusUpdater 0x55594d132360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.271+0000 7fb8e3b8e640 -1 rbd::mirror::NamespaceReplayer: 0x55594d0dc820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.271+0000 7fb8e3b8e640 -1 rbd::mirror::NamespaceReplayer: 0x55594d0dc9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.271+0000 7fb8ddb82640 -1 rbd::mirror::MirrorStatusUpdater 0x55594d132000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.271+0000 7fb8e438f640 -1 rbd::mirror::NamespaceReplayer: 0x55594d0dc820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.271+0000 7fb8dd381640 -1 rbd::mirror::MirrorStatusUpdater 0x55594d132120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.271+0000 7fb8e3b8e640 -1 rbd::mirror::NamespaceReplayer: 0x55594d0dc9c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.272+0000 7fb8dd381640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55594d0b6fc0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-13T10:01:44.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.273+0000 7fb8ddb82640 -1 rbd::mirror::PoolWatcher: 0x55594d71f860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-13T10:01:44.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.273+0000 7fb8d7375640 -1 rbd::mirror::PoolWatcher: 0x55594d7a5860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-13T10:01:44.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.273+0000 7fb8e438f640 -1 rbd::mirror::LeaderWatcher: 0x55594ceb1b00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-13T10:01:44.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.273+0000 7fb8e438f640 -1 rbd::mirror::LeaderWatcher: 0x55594ceb1b00 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-13T10:01:44.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.273+0000 7fb8e3b8e640 -1 rbd::mirror::LeaderWatcher: 0x55594ceb1b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-13T10:01:44.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.273+0000 7fb8e3b8e640 -1 rbd::mirror::InstanceWatcher: 0x55594d0e2000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5722 object: (2) No such file or directory 2024-09-13T10:01:44.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.274+0000 7fb8dd381640 -1 rbd::mirror::InstanceWatcher: 0x55594d0e2000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.274+0000 7fb8d6b74640 -1 rbd::mirror::MirrorStatusUpdater 0x55594cec25a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.274+0000 7fb8e3b8e640 -1 rbd::mirror::NamespaceReplayer: 0x55594d0dc000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.274+0000 7fb8dd381640 -1 rbd::mirror::MirrorStatusUpdater 0x55594ce5c7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.274+0000 7fb8e438f640 -1 rbd::mirror::NamespaceReplayer: 0x55594d0dc000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2024-09-13T10:01:44.287 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-13T10:01:44.344 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2024-09-13T10:01:44.344 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T10:01:44.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.334+0000 7f66320aa640 -1 rbd::mirror::RemotePoolPoller: 0x5599d2179790 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-09-13T10:01:44.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.334+0000 7f66320aa640 -1 rbd::mirror::RemotePoolPoller: 0x5599d2179790 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-09-13T10:01:44.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.334+0000 7f66320aa640 -1 rbd::mirror::PoolReplayer: 0x5599d332a000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-09-13T10:01:44.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.343+0000 7f9fa9dec640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 71829) UID: 1000 2024-09-13T10:01:44.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.344+0000 7f9fa8dea640 -1 rbd::mirror::InstanceWatcher: 0x563dad064700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6637 object: (2) No such file or directory 2024-09-13T10:01:44.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.344+0000 7f9fa2dde640 -1 rbd::mirror::InstanceWatcher: 0x563dad064380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.344+0000 7f9f94dc2640 -1 rbd::mirror::InstanceWatcher: 0x563dad588000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.344+0000 7f9fa2dde640 -1 rbd::mirror::InstanceWatcher: 0x563dad064700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.345+0000 7f9f9c5d1640 -1 rbd::mirror::MirrorStatusUpdater 0x563dad0ae5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.345+0000 7f9f9c5d1640 -1 rbd::mirror::MirrorStatusUpdater 0x563dad0ae6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.345+0000 7f9fa8dea640 -1 rbd::mirror::NamespaceReplayer: 0x563dad05c820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.345+0000 7f9fa8dea640 -1 rbd::mirror::NamespaceReplayer: 0x563dad05c9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.345+0000 7f9fa2dde640 -1 rbd::mirror::MirrorStatusUpdater 0x563dad0ae240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.345+0000 7f9fa95eb640 -1 rbd::mirror::NamespaceReplayer: 0x563dad05c820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.346+0000 7f9fa95eb640 -1 rbd::mirror::LeaderWatcher: 0x563dace49b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-13T10:01:44.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.347+0000 7f9fa95eb640 -1 rbd::mirror::InstanceWatcher: 0x563dad064000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6637 object: (2) No such file or directory 2024-09-13T10:01:44.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.347+0000 7f9fa25dd640 -1 rbd::mirror::InstanceWatcher: 0x563dad064000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.347+0000 7f9f9c5d1640 -1 rbd::mirror::MirrorStatusUpdater 0x563dace366c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.347+0000 7f9fa95eb640 -1 rbd::mirror::NamespaceReplayer: 0x563dad05c000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.347+0000 7f9fa25dd640 -1 rbd::mirror::MirrorStatusUpdater 0x563dacd8e7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.347+0000 7f9fa8dea640 -1 rbd::mirror::NamespaceReplayer: 0x563dad05c000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2024-09-13T10:01:44.359 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-13T10:01:44.411 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2024-09-13T10:01:44.411 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T10:01:44.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.404+0000 7f664c8df640 -1 rbd::mirror::LeaderWatcher: 0x5599d30efb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-13T10:01:44.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.404+0000 7f664c8df640 -1 rbd::mirror::InstanceWatcher: 0x5599d3322000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6068 object: (2) No such file or directory 2024-09-13T10:01:44.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.404+0000 7f66460d2640 -1 rbd::mirror::InstanceWatcher: 0x5599d3322000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.405+0000 7f66468d3640 -1 rbd::mirror::MirrorStatusUpdater 0x5599d30587e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.405+0000 7f664c8df640 -1 rbd::mirror::NamespaceReplayer: 0x5599d3316000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.407+0000 7f66388b7640 -1 rbd::mirror::InstanceWatcher: 0x5599d3323c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.410+0000 7fb789d51640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 54328) UID: 1000 2024-09-13T10:01:44.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.411+0000 7fb775528640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55f26445ad80 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-13T10:01:44.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.411+0000 7fb789550640 -1 rbd::mirror::InstanceWatcher: 0x55f2644d0380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5528 object: (2) No such file or directory 2024-09-13T10:01:44.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.411+0000 7fb789550640 -1 rbd::mirror::InstanceWatcher: 0x55f2644d0700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5528 object: (2) No such file or directory 2024-09-13T10:01:44.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.411+0000 7fb775528640 -1 rbd::mirror::PoolWatcher: 0x55f264ac63c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-13T10:01:44.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.411+0000 7fb782d43640 -1 rbd::mirror::InstanceWatcher: 0x55f2644d0380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.411+0000 7fb782542640 -1 rbd::mirror::InstanceWatcher: 0x55f2644d0700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.411+0000 7fb76ed1b640 -1 rbd::mirror::PoolWatcher: 0x55f264ac6d20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-13T10:01:44.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.412+0000 7fb77bd35640 -1 rbd::mirror::MirrorStatusUpdater 0x55f2645185a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.412+0000 7fb77bd35640 -1 rbd::mirror::MirrorStatusUpdater 0x55f264518360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.412+0000 7fb788d4f640 -1 rbd::mirror::NamespaceReplayer: 0x55f2644bc820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.412+0000 7fb788d4f640 -1 rbd::mirror::NamespaceReplayer: 0x55f2644bc9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.412+0000 7fb789550640 -1 rbd::mirror::LeaderWatcher: 0x55f2644a1500 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-13T10:01:44.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.412+0000 7fb782d43640 -1 rbd::mirror::MirrorStatusUpdater 0x55f264518000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.412+0000 7fb789550640 -1 rbd::mirror::LeaderWatcher: 0x55f2644a1500 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-13T10:01:44.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.412+0000 7fb789550640 -1 rbd::mirror::NamespaceReplayer: 0x55f2644bc820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.412+0000 7fb788d4f640 -1 rbd::mirror::LeaderWatcher: 0x55f2644a1500 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-13T10:01:44.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.412+0000 7fb788d4f640 -1 rbd::mirror::InstanceWatcher: 0x55f2644d0a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5531 object: (2) No such file or directory 2024-09-13T10:01:44.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.412+0000 7fb774d27640 -1 rbd::mirror::InstanceWatcher: 0x55f2644d0a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.413+0000 7fb76ed1b640 -1 rbd::mirror::MirrorStatusUpdater 0x55f2647586c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.413+0000 7fb788d4f640 -1 rbd::mirror::NamespaceReplayer: 0x55f2648dc4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.413+0000 7fb774d27640 -1 rbd::mirror::MirrorStatusUpdater 0x55f26471c240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.413+0000 7fb789550640 -1 rbd::mirror::NamespaceReplayer: 0x55f2648dc4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.415+0000 7fb788d4f640 -1 rbd::mirror::LeaderWatcher: 0x55f264296c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-13T10:01:44.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.415+0000 7fb788d4f640 -1 rbd::mirror::InstanceWatcher: 0x55f2644d0000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5528 object: (2) No such file or directory 2024-09-13T10:01:44.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.415+0000 7fb782542640 -1 rbd::mirror::InstanceWatcher: 0x55f2644d0000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.416+0000 7fb77c536640 -1 rbd::mirror::MirrorStatusUpdater 0x55f264242d80 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.416+0000 7fb788d4f640 -1 rbd::mirror::NamespaceReplayer: 0x55f2644bc000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.416+0000 7fb782d43640 -1 rbd::mirror::MirrorStatusUpdater 0x55f264243320 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.416+0000 7fb789550640 -1 rbd::mirror::NamespaceReplayer: 0x55f2644bc000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2024-09-13T10:01:44.429 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-13T10:01:44.444 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2024-09-13T10:01:44.444 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T10:01:44.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.469+0000 7fdb54ee8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 71189) UID: 1000 2024-09-13T10:01:44.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2024-09-13T10:01:44.481 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-13T10:01:44.492 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2024-09-13T10:01:44.492 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T10:01:44.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.521+0000 7f7193ec9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 62026) UID: 1000 2024-09-13T10:01:44.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.522+0000 7f71936c8640 -1 rbd::mirror::LeaderWatcher: 0x559eec468300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-13T10:01:44.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.522+0000 7f7192ec7640 -1 rbd::mirror::InstanceWatcher: 0x559eec04c380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5624 object: (2) No such file or directory 2024-09-13T10:01:44.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.522+0000 7f71936c8640 -1 rbd::mirror::InstanceWatcher: 0x559eec04c700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5624 object: (2) No such file or directory 2024-09-13T10:01:44.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.522+0000 7f71936c8640 -1 rbd::mirror::InstanceWatcher: 0x559eec04ca80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5627 object: (2) No such file or directory 2024-09-13T10:01:44.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.522+0000 7f718c6ba640 -1 rbd::mirror::InstanceWatcher: 0x559eec04c380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.522+0000 7f718cebb640 -1 rbd::mirror::InstanceWatcher: 0x559eec04c700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.523+0000 7f718c6ba640 -1 rbd::mirror::MirrorStatusUpdater 0x559eec090120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.523+0000 7f71936c8640 -1 rbd::mirror::NamespaceReplayer: 0x559eec0429c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.523+0000 7f717f6a0640 -1 rbd::mirror::InstanceWatcher: 0x559eec04ca80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.523+0000 7f7185ead640 -1 rbd::mirror::MirrorStatusUpdater 0x559eec090360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.523+0000 7f7192ec7640 -1 rbd::mirror::NamespaceReplayer: 0x559eec042820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.523+0000 7f718cebb640 -1 rbd::mirror::MirrorStatusUpdater 0x559eec090000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.523+0000 7f71936c8640 -1 rbd::mirror::NamespaceReplayer: 0x559eec042820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.523+0000 7f7178692640 -1 rbd::mirror::MirrorStatusUpdater 0x559eec2a47e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.523+0000 7f7192ec7640 -1 rbd::mirror::NamespaceReplayer: 0x559eec3c8680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.523+0000 7f717ee9f640 -1 rbd::mirror::MirrorStatusUpdater 0x559eec270120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.523+0000 7f71936c8640 -1 rbd::mirror::NamespaceReplayer: 0x559eec3c8680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.525+0000 7f7192ec7640 -1 rbd::mirror::LeaderWatcher: 0x559eebe0db00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-13T10:01:44.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.525+0000 7f7192ec7640 -1 rbd::mirror::InstanceWatcher: 0x559eec04c000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5624 object: (2) No such file or directory 2024-09-13T10:01:44.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.526+0000 7f718cebb640 -1 rbd::mirror::InstanceWatcher: 0x559eec04c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.526+0000 7f7185ead640 -1 rbd::mirror::MirrorStatusUpdater 0x559eebdf66c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.526+0000 7f7192ec7640 -1 rbd::mirror::NamespaceReplayer: 0x559eec042000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.526+0000 7f718c6ba640 -1 rbd::mirror::MirrorStatusUpdater 0x559eebd9a6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi190.stderr:2024-09-13T10:01:44.526+0000 7f71936c8640 -1 rbd::mirror::NamespaceReplayer: 0x559eec042000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2024-09-13T10:01:44.540 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-13T10:01:44.551 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2024-09-13T10:01:44.551 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T10:01:44.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.580+0000 7f717fdd7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 45486) UID: 1000 2024-09-13T10:01:44.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.581+0000 7f717f5d6640 -1 rbd::mirror::LeaderWatcher: 0x55f300f9a300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-13T10:01:44.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.581+0000 7f717f5d6640 -1 rbd::mirror::InstanceWatcher: 0x55f300b3e380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4738 object: (2) No such file or directory 2024-09-13T10:01:44.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.581+0000 7f717f5d6640 -1 rbd::mirror::InstanceWatcher: 0x55f301074000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4738 object: (2) No such file or directory 2024-09-13T10:01:44.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.581+0000 7f717f5d6640 -1 rbd::mirror::InstanceWatcher: 0x55f300b3fc00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4741 object: (2) No such file or directory 2024-09-13T10:01:44.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.581+0000 7f71785c8640 -1 rbd::mirror::InstanceWatcher: 0x55f300b3e380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.581+0000 7f71785c8640 -1 rbd::mirror::InstanceWatcher: 0x55f301074000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.581+0000 7f716b5ae640 -1 rbd::mirror::InstanceWatcher: 0x55f300b3fc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.582+0000 7f71725bc640 -1 rbd::mirror::MirrorStatusUpdater 0x55f300bb8360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.582+0000 7f71725bc640 -1 rbd::mirror::MirrorStatusUpdater 0x55f300bb85a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.582+0000 7f717edd5640 -1 rbd::mirror::NamespaceReplayer: 0x55f300b36820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.582+0000 7f717edd5640 -1 rbd::mirror::NamespaceReplayer: 0x55f300b369c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.582+0000 7f71785c8640 -1 rbd::mirror::MirrorStatusUpdater 0x55f300bb8000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.582+0000 7f71785c8640 -1 rbd::mirror::MirrorStatusUpdater 0x55f300bb8120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.582+0000 7f71645a0640 -1 rbd::mirror::MirrorStatusUpdater 0x55f300bb86c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.582+0000 7f717f5d6640 -1 rbd::mirror::NamespaceReplayer: 0x55f300b36820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.582+0000 7f717f5d6640 -1 rbd::mirror::NamespaceReplayer: 0x55f300b369c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.582+0000 7f717f5d6640 -1 rbd::mirror::NamespaceReplayer: 0x55f300fa04e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.582+0000 7f716adad640 -1 rbd::mirror::MirrorStatusUpdater 0x55f300d8c360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.582+0000 7f717edd5640 -1 rbd::mirror::NamespaceReplayer: 0x55f300fa04e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.584+0000 7f717f5d6640 -1 rbd::mirror::LeaderWatcher: 0x55f300909b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-13T10:01:44.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.584+0000 7f717f5d6640 -1 rbd::mirror::InstanceWatcher: 0x55f300b3e000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4738 object: (2) No such file or directory 2024-09-13T10:01:44.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.585+0000 7f7178dc9640 -1 rbd::mirror::InstanceWatcher: 0x55f300b3e000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.585+0000 7f7171dbb640 -1 rbd::mirror::MirrorStatusUpdater 0x55f3008ec6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.585+0000 7f717f5d6640 -1 rbd::mirror::NamespaceReplayer: 0x55f300b36000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.585+0000 7f71785c8640 -1 rbd::mirror::MirrorStatusUpdater 0x55f3008727e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi190.stderr:2024-09-13T10:01:44.585+0000 7f717edd5640 -1 rbd::mirror::NamespaceReplayer: 0x55f300b36000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2024-09-13T10:01:44.600 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-13T10:01:44.612 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2024-09-13T10:01:44.612 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T10:01:44.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.640+0000 7f890904c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 60184) UID: 1000 2024-09-13T10:01:44.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.641+0000 7f88f4823640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x557705cd3200 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-13T10:01:44.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.641+0000 7f890804a640 -1 rbd::mirror::InstanceWatcher: 0x557702334380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5501 object: (2) No such file or directory 2024-09-13T10:01:44.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.641+0000 7f890884b640 -1 rbd::mirror::InstanceWatcher: 0x557702334700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5501 object: (2) No such file or directory 2024-09-13T10:01:44.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.641+0000 7f88f4823640 -1 rbd::mirror::PoolWatcher: 0x5577033c61e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-13T10:01:44.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.641+0000 7f88ee016640 -1 rbd::mirror::PoolWatcher: 0x5577033c6000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-13T10:01:44.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.641+0000 7f890183d640 -1 rbd::mirror::InstanceWatcher: 0x557702334380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.641+0000 7f890203e640 -1 rbd::mirror::InstanceWatcher: 0x557702334700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.641+0000 7f88fb831640 -1 rbd::mirror::MirrorStatusUpdater 0x55770230c360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.641+0000 7f88fb831640 -1 rbd::mirror::MirrorStatusUpdater 0x55770230c5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.641+0000 7f890804a640 -1 rbd::mirror::NamespaceReplayer: 0x55770232a820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.641+0000 7f890804a640 -1 rbd::mirror::NamespaceReplayer: 0x55770232a9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.641+0000 7f890804a640 -1 rbd::mirror::LeaderWatcher: 0x557702734300 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-13T10:01:44.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.642+0000 7f890804a640 -1 rbd::mirror::LeaderWatcher: 0x557702734300 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-13T10:01:44.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.642+0000 7f890183d640 -1 rbd::mirror::MirrorStatusUpdater 0x55770230c000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.642+0000 7f890183d640 -1 rbd::mirror::MirrorStatusUpdater 0x55770230c120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.642+0000 7f890884b640 -1 rbd::mirror::NamespaceReplayer: 0x55770232a820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.642+0000 7f890884b640 -1 rbd::mirror::NamespaceReplayer: 0x55770232a9c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.642+0000 7f890804a640 -1 rbd::mirror::LeaderWatcher: 0x557702734300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-13T10:01:44.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.642+0000 7f890804a640 -1 rbd::mirror::InstanceWatcher: 0x5577027e8000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5504 object: (2) No such file or directory 2024-09-13T10:01:44.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.642+0000 7f88f4823640 -1 rbd::mirror::InstanceWatcher: 0x5577027e8000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.642+0000 7f88ee016640 -1 rbd::mirror::MirrorStatusUpdater 0x55770230cb40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.642+0000 7f890804a640 -1 rbd::mirror::NamespaceReplayer: 0x557702696680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.643+0000 7f88f4823640 -1 rbd::mirror::MirrorStatusUpdater 0x5577025425a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.643+0000 7f890884b640 -1 rbd::mirror::NamespaceReplayer: 0x557702696680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.645+0000 7f890804a640 -1 rbd::mirror::LeaderWatcher: 0x5577020f9b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-13T10:01:44.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.645+0000 7f890804a640 -1 rbd::mirror::InstanceWatcher: 0x557702334000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5501 object: (2) No such file or directory 2024-09-13T10:01:44.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.646+0000 7f890183d640 -1 rbd::mirror::InstanceWatcher: 0x557702334000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-13T10:01:44.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.646+0000 7f88fb030640 -1 rbd::mirror::MirrorStatusUpdater 0x5577020e26c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.646+0000 7f890804a640 -1 rbd::mirror::NamespaceReplayer: 0x55770232a000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.646+0000 7f890203e640 -1 rbd::mirror::MirrorStatusUpdater 0x5577020847e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-13T10:01:44.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi190.stderr:2024-09-13T10:01:44.646+0000 7f890884b640 -1 rbd::mirror::NamespaceReplayer: 0x55770232a000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-13T10:01:44.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2024-09-13T10:01:44.660 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-13T10:01:44.671 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2024-09-13T10:01:44.671 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T10:01:44.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi190.stderr:2024-09-13T10:01:44.700+0000 7f664d8e1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 70326) UID: 1000 2024-09-13T10:01:44.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2024-09-13T10:01:44.710 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-13T10:01:44.722 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-13T10:01:44.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-13T10:01:45.038 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T10:01:45.038 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2024-09-13T10:01:45.050 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1598,"stamp":"2024-09-13T10:01:43.776627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":227,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4362244,"kb_used_data":15972,"kb_used_omap":4,"kb_used_meta":4346235,"kb_avail":278753276,"statfs":{"total":289910292480,"available":285443354624,"internally_reserved":0,"allocated":16355328,"data_stored":9122519,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":4450544978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536816694,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-117898,"num_read_kb":-39655415,"num_write":-625272,"num_write_kb":-231631040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-10,"num_bytes_recovered":-25165824,"num_keys_recovered":-4,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001164"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":474,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.305791+0000","last_change":"2024-09-13T09:23:52.448692+0000","last_active":"2024-09-13T10:01:39.305791+0000","last_peered":"2024-09-13T10:01:39.305791+0000","last_clean":"2024-09-13T10:01:39.305791+0000","last_became_active":"2024-09-13T09:23:50.442738+0000","last_became_peered":"2024-09-13T09:23:50.442738+0000","last_unstale":"2024-09-13T10:01:39.305791+0000","last_undegraded":"2024-09-13T10:01:39.305791+0000","last_fullsized":"2024-09-13T10:01:39.305791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:00:17.513367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072094900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":474,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.306055+0000","last_change":"2024-09-13T09:23:52.448736+0000","last_active":"2024-09-13T10:01:39.306055+0000","last_peered":"2024-09-13T10:01:39.306055+0000","last_clean":"2024-09-13T10:01:39.306055+0000","last_became_active":"2024-09-13T09:23:50.443108+0000","last_became_peered":"2024-09-13T09:23:50.443108+0000","last_unstale":"2024-09-13T10:01:39.306055+0000","last_undegraded":"2024-09-13T10:01:39.306055+0000","last_fullsized":"2024-09-13T10:01:39.306055+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:24.324564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064757699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":475,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.306453+0000","last_change":"2024-09-13T09:23:52.448503+0000","last_active":"2024-09-13T10:01:39.306453+0000","last_peered":"2024-09-13T10:01:39.306453+0000","last_clean":"2024-09-13T10:01:39.306453+0000","last_became_active":"2024-09-13T09:23:50.439969+0000","last_became_peered":"2024-09-13T09:23:50.439969+0000","last_unstale":"2024-09-13T10:01:39.306453+0000","last_undegraded":"2024-09-13T10:01:39.306453+0000","last_fullsized":"2024-09-13T10:01:39.306453+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:50.713752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052999299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":475,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.305027+0000","last_change":"2024-09-13T09:23:52.448963+0000","last_active":"2024-09-13T10:01:39.305027+0000","last_peered":"2024-09-13T10:01:39.305027+0000","last_clean":"2024-09-13T10:01:39.305027+0000","last_became_active":"2024-09-13T09:23:50.442812+0000","last_became_peered":"2024-09-13T09:23:50.442812+0000","last_unstale":"2024-09-13T10:01:39.305027+0000","last_undegraded":"2024-09-13T10:01:39.305027+0000","last_fullsized":"2024-09-13T10:01:39.305027+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:43.338946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088972200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":477,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.315618+0000","last_change":"2024-09-13T09:23:52.446617+0000","last_active":"2024-09-13T10:01:39.315618+0000","last_peered":"2024-09-13T10:01:39.315618+0000","last_clean":"2024-09-13T10:01:39.315618+0000","last_became_active":"2024-09-13T09:23:50.442111+0000","last_became_peered":"2024-09-13T09:23:50.442111+0000","last_unstale":"2024-09-13T10:01:39.315618+0000","last_undegraded":"2024-09-13T10:01:39.315618+0000","last_fullsized":"2024-09-13T10:01:39.315618+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:15.440943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000891642,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":474,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.309091+0000","last_change":"2024-09-13T09:23:52.443957+0000","last_active":"2024-09-13T10:01:39.309091+0000","last_peered":"2024-09-13T10:01:39.309091+0000","last_clean":"2024-09-13T10:01:39.309091+0000","last_became_active":"2024-09-13T09:23:50.441154+0000","last_became_peered":"2024-09-13T09:23:50.441154+0000","last_unstale":"2024-09-13T10:01:39.309091+0000","last_undegraded":"2024-09-13T10:01:39.309091+0000","last_fullsized":"2024-09-13T10:01:39.309091+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:40.913446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033978799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":474,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.310150+0000","last_change":"2024-09-13T09:23:52.444000+0000","last_active":"2024-09-13T10:01:39.310150+0000","last_peered":"2024-09-13T10:01:39.310150+0000","last_clean":"2024-09-13T10:01:39.310150+0000","last_became_active":"2024-09-13T09:23:50.441634+0000","last_became_peered":"2024-09-13T09:23:50.441634+0000","last_unstale":"2024-09-13T10:01:39.310150+0000","last_undegraded":"2024-09-13T10:01:39.310150+0000","last_fullsized":"2024-09-13T10:01:39.310150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:10.844086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038734700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":476,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.305125+0000","last_change":"2024-09-13T09:23:52.448440+0000","last_active":"2024-09-13T10:01:39.305125+0000","last_peered":"2024-09-13T10:01:39.305125+0000","last_clean":"2024-09-13T10:01:39.305125+0000","last_became_active":"2024-09-13T09:23:50.440112+0000","last_became_peered":"2024-09-13T09:23:50.440112+0000","last_unstale":"2024-09-13T10:01:39.305125+0000","last_undegraded":"2024-09-13T10:01:39.305125+0000","last_fullsized":"2024-09-13T10:01:39.305125+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:40.210218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048773600000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":600,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.304895+0000","last_change":"2024-09-13T09:23:45.887443+0000","last_active":"2024-09-13T10:01:39.304895+0000","last_peered":"2024-09-13T10:01:39.304895+0000","last_clean":"2024-09-13T10:01:39.304895+0000","last_became_active":"2024-09-13T09:23:45.886500+0000","last_became_peered":"2024-09-13T09:23:45.886500+0000","last_unstale":"2024-09-13T10:01:39.304895+0000","last_undegraded":"2024-09-13T10:01:39.304895+0000","last_fullsized":"2024-09-13T10:01:39.304895+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:44.266705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:44.266705+0000","last_clean_scrub_stamp":"2024-09-13T09:23:44.266705+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:28.855304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738831,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1300468,"kb_used_data":5288,"kb_used_omap":1,"kb_used_meta":1295166,"kb_avail":93071372,"statfs":{"total":96636764160,"available":95305084928,"internally_reserved":0,"allocated":5414912,"data_stored":2649626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1326250438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.69299999999999995,"15min":0.46500000000000002},"min":{"1min":0.182,"5min":0.14799999999999999,"15min":0.13700000000000001},"max":{"1min":7.9580000000000002,"5min":10.542,"15min":10.542},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.76400000000000001,"15min":0.53000000000000003},"min":{"1min":0.23599999999999999,"5min":0.19800000000000001,"15min":0.14299999999999999},"max":{"1min":7.9909999999999997,"5min":10.577999999999999,"15min":10.577999999999999},"last":0.40100000000000002}]},{"osd":1,"last update":"Fri Sep 13 10:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.48999999999999999,"15min":0.42499999999999999},"min":{"1min":0.19700000000000001,"5min":0.183,"15min":0.16},"max":{"1min":4.4210000000000003,"5min":4.4210000000000003,"15min":4.6429999999999998},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.115,"5min":0.85199999999999998,"15min":0.56299999999999994},"min":{"1min":0.219,"5min":0.193,"15min":0.16900000000000001},"max":{"1min":9.5009999999999994,"5min":9.8219999999999992,"15min":9.8219999999999992},"last":0.38600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738827,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1441504,"kb_used_data":5336,"kb_used_omap":1,"kb_used_meta":1436158,"kb_avail":92930336,"statfs":{"total":96636764160,"available":95160664064,"internally_reserved":0,"allocated":5464064,"data_stored":3232899,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1470626246},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:00:41 2024","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":0.84999999999999998,"15min":0.51500000000000001},"min":{"1min":0.23699999999999999,"5min":0.14299999999999999,"15min":0.13500000000000001},"max":{"1min":11.082000000000001,"5min":11.082000000000001,"15min":11.082000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63400000000000001,"15min":0.46700000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.17599999999999999},"max":{"1min":1.754,"5min":6.8369999999999997,"15min":6.9790000000000001},"last":0.35999999999999999}]},{"osd":2,"last update":"Fri Sep 13 10:00:41 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":0.73299999999999998,"15min":0.48899999999999999},"min":{"1min":0.20300000000000001,"5min":0.186,"15min":0.16900000000000001},"max":{"1min":11.034000000000001,"5min":11.034000000000001,"15min":11.034000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54800000000000004,"15min":0.40799999999999997},"min":{"1min":0.217,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.673,"5min":3.835,"15min":3.835},"last":0.312}]}]},{"osd":0,"up_from":8,"seq":34359738829,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1620272,"kb_used_data":5348,"kb_used_omap":1,"kb_used_meta":1614910,"kb_avail":92751568,"statfs":{"total":96636764160,"available":94977605632,"internally_reserved":0,"allocated":5476352,"data_stored":3239994,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1653668294},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 10:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.748,"15min":0.47299999999999998},"min":{"1min":0.19,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":7.5960000000000001,"5min":10.384,"15min":10.384},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.80800000000000005,"15min":0.52300000000000002},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.17799999999999999},"max":{"1min":7.6340000000000003,"5min":10.427,"15min":10.427},"last":0.25700000000000001}]},{"osd":2,"last update":"Fri Sep 13 10:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.91900000000000004,"15min":0.53400000000000003},"min":{"1min":0.20599999999999999,"5min":0.17999999999999999,"15min":0.16200000000000001},"max":{"1min":6.6280000000000001,"5min":14.365,"15min":14.365},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.98099999999999998,"15min":0.56000000000000005},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19700000000000001},"max":{"1min":6.6829999999999998,"5min":14.401,"15min":14.401},"last":0.27000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T10:01:45.052 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-13T10:01:45.317 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T10:01:45.317 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2024-09-13T10:01:45.329 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1598,"stamp":"2024-09-13T10:01:43.776627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":227,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4362244,"kb_used_data":15972,"kb_used_omap":4,"kb_used_meta":4346235,"kb_avail":278753276,"statfs":{"total":289910292480,"available":285443354624,"internally_reserved":0,"allocated":16355328,"data_stored":9122519,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":4450544978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536816694,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-117898,"num_read_kb":-39655415,"num_write":-625272,"num_write_kb":-231631040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-10,"num_bytes_recovered":-25165824,"num_keys_recovered":-4,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001164"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":474,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.305791+0000","last_change":"2024-09-13T09:23:52.448692+0000","last_active":"2024-09-13T10:01:39.305791+0000","last_peered":"2024-09-13T10:01:39.305791+0000","last_clean":"2024-09-13T10:01:39.305791+0000","last_became_active":"2024-09-13T09:23:50.442738+0000","last_became_peered":"2024-09-13T09:23:50.442738+0000","last_unstale":"2024-09-13T10:01:39.305791+0000","last_undegraded":"2024-09-13T10:01:39.305791+0000","last_fullsized":"2024-09-13T10:01:39.305791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:00:17.513367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072094900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":474,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.306055+0000","last_change":"2024-09-13T09:23:52.448736+0000","last_active":"2024-09-13T10:01:39.306055+0000","last_peered":"2024-09-13T10:01:39.306055+0000","last_clean":"2024-09-13T10:01:39.306055+0000","last_became_active":"2024-09-13T09:23:50.443108+0000","last_became_peered":"2024-09-13T09:23:50.443108+0000","last_unstale":"2024-09-13T10:01:39.306055+0000","last_undegraded":"2024-09-13T10:01:39.306055+0000","last_fullsized":"2024-09-13T10:01:39.306055+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:24.324564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064757699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":475,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.306453+0000","last_change":"2024-09-13T09:23:52.448503+0000","last_active":"2024-09-13T10:01:39.306453+0000","last_peered":"2024-09-13T10:01:39.306453+0000","last_clean":"2024-09-13T10:01:39.306453+0000","last_became_active":"2024-09-13T09:23:50.439969+0000","last_became_peered":"2024-09-13T09:23:50.439969+0000","last_unstale":"2024-09-13T10:01:39.306453+0000","last_undegraded":"2024-09-13T10:01:39.306453+0000","last_fullsized":"2024-09-13T10:01:39.306453+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:50.713752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052999299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":475,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.305027+0000","last_change":"2024-09-13T09:23:52.448963+0000","last_active":"2024-09-13T10:01:39.305027+0000","last_peered":"2024-09-13T10:01:39.305027+0000","last_clean":"2024-09-13T10:01:39.305027+0000","last_became_active":"2024-09-13T09:23:50.442812+0000","last_became_peered":"2024-09-13T09:23:50.442812+0000","last_unstale":"2024-09-13T10:01:39.305027+0000","last_undegraded":"2024-09-13T10:01:39.305027+0000","last_fullsized":"2024-09-13T10:01:39.305027+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:43.338946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088972200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":477,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.315618+0000","last_change":"2024-09-13T09:23:52.446617+0000","last_active":"2024-09-13T10:01:39.315618+0000","last_peered":"2024-09-13T10:01:39.315618+0000","last_clean":"2024-09-13T10:01:39.315618+0000","last_became_active":"2024-09-13T09:23:50.442111+0000","last_became_peered":"2024-09-13T09:23:50.442111+0000","last_unstale":"2024-09-13T10:01:39.315618+0000","last_undegraded":"2024-09-13T10:01:39.315618+0000","last_fullsized":"2024-09-13T10:01:39.315618+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:15.440943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000891642,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":474,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.309091+0000","last_change":"2024-09-13T09:23:52.443957+0000","last_active":"2024-09-13T10:01:39.309091+0000","last_peered":"2024-09-13T10:01:39.309091+0000","last_clean":"2024-09-13T10:01:39.309091+0000","last_became_active":"2024-09-13T09:23:50.441154+0000","last_became_peered":"2024-09-13T09:23:50.441154+0000","last_unstale":"2024-09-13T10:01:39.309091+0000","last_undegraded":"2024-09-13T10:01:39.309091+0000","last_fullsized":"2024-09-13T10:01:39.309091+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:40.913446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033978799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":474,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.310150+0000","last_change":"2024-09-13T09:23:52.444000+0000","last_active":"2024-09-13T10:01:39.310150+0000","last_peered":"2024-09-13T10:01:39.310150+0000","last_clean":"2024-09-13T10:01:39.310150+0000","last_became_active":"2024-09-13T09:23:50.441634+0000","last_became_peered":"2024-09-13T09:23:50.441634+0000","last_unstale":"2024-09-13T10:01:39.310150+0000","last_undegraded":"2024-09-13T10:01:39.310150+0000","last_fullsized":"2024-09-13T10:01:39.310150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:10.844086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038734700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":476,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.305125+0000","last_change":"2024-09-13T09:23:52.448440+0000","last_active":"2024-09-13T10:01:39.305125+0000","last_peered":"2024-09-13T10:01:39.305125+0000","last_clean":"2024-09-13T10:01:39.305125+0000","last_became_active":"2024-09-13T09:23:50.440112+0000","last_became_peered":"2024-09-13T09:23:50.440112+0000","last_unstale":"2024-09-13T10:01:39.305125+0000","last_undegraded":"2024-09-13T10:01:39.305125+0000","last_fullsized":"2024-09-13T10:01:39.305125+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:40.210218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048773600000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":600,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.304895+0000","last_change":"2024-09-13T09:23:45.887443+0000","last_active":"2024-09-13T10:01:39.304895+0000","last_peered":"2024-09-13T10:01:39.304895+0000","last_clean":"2024-09-13T10:01:39.304895+0000","last_became_active":"2024-09-13T09:23:45.886500+0000","last_became_peered":"2024-09-13T09:23:45.886500+0000","last_unstale":"2024-09-13T10:01:39.304895+0000","last_undegraded":"2024-09-13T10:01:39.304895+0000","last_fullsized":"2024-09-13T10:01:39.304895+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:44.266705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:44.266705+0000","last_clean_scrub_stamp":"2024-09-13T09:23:44.266705+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:28.855304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738831,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1300468,"kb_used_data":5288,"kb_used_omap":1,"kb_used_meta":1295166,"kb_avail":93071372,"statfs":{"total":96636764160,"available":95305084928,"internally_reserved":0,"allocated":5414912,"data_stored":2649626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1326250438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.69299999999999995,"15min":0.46500000000000002},"min":{"1min":0.182,"5min":0.14799999999999999,"15min":0.13700000000000001},"max":{"1min":7.9580000000000002,"5min":10.542,"15min":10.542},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.76400000000000001,"15min":0.53000000000000003},"min":{"1min":0.23599999999999999,"5min":0.19800000000000001,"15min":0.14299999999999999},"max":{"1min":7.9909999999999997,"5min":10.577999999999999,"15min":10.577999999999999},"last":0.40100000000000002}]},{"osd":1,"last update":"Fri Sep 13 10:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.48999999999999999,"15min":0.42499999999999999},"min":{"1min":0.19700000000000001,"5min":0.183,"15min":0.16},"max":{"1min":4.4210000000000003,"5min":4.4210000000000003,"15min":4.6429999999999998},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.115,"5min":0.85199999999999998,"15min":0.56299999999999994},"min":{"1min":0.219,"5min":0.193,"15min":0.16900000000000001},"max":{"1min":9.5009999999999994,"5min":9.8219999999999992,"15min":9.8219999999999992},"last":0.38600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738827,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1441504,"kb_used_data":5336,"kb_used_omap":1,"kb_used_meta":1436158,"kb_avail":92930336,"statfs":{"total":96636764160,"available":95160664064,"internally_reserved":0,"allocated":5464064,"data_stored":3232899,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1470626246},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:00:41 2024","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":0.84999999999999998,"15min":0.51500000000000001},"min":{"1min":0.23699999999999999,"5min":0.14299999999999999,"15min":0.13500000000000001},"max":{"1min":11.082000000000001,"5min":11.082000000000001,"15min":11.082000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63400000000000001,"15min":0.46700000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.17599999999999999},"max":{"1min":1.754,"5min":6.8369999999999997,"15min":6.9790000000000001},"last":0.35999999999999999}]},{"osd":2,"last update":"Fri Sep 13 10:00:41 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":0.73299999999999998,"15min":0.48899999999999999},"min":{"1min":0.20300000000000001,"5min":0.186,"15min":0.16900000000000001},"max":{"1min":11.034000000000001,"5min":11.034000000000001,"15min":11.034000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54800000000000004,"15min":0.40799999999999997},"min":{"1min":0.217,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.673,"5min":3.835,"15min":3.835},"last":0.312}]}]},{"osd":0,"up_from":8,"seq":34359738829,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1620272,"kb_used_data":5348,"kb_used_omap":1,"kb_used_meta":1614910,"kb_avail":92751568,"statfs":{"total":96636764160,"available":94977605632,"internally_reserved":0,"allocated":5476352,"data_stored":3239994,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1653668294},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 10:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.748,"15min":0.47299999999999998},"min":{"1min":0.19,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":7.5960000000000001,"5min":10.384,"15min":10.384},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.80800000000000005,"15min":0.52300000000000002},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.17799999999999999},"max":{"1min":7.6340000000000003,"5min":10.427,"15min":10.427},"last":0.25700000000000001}]},{"osd":2,"last update":"Fri Sep 13 10:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.91900000000000004,"15min":0.53400000000000003},"min":{"1min":0.20599999999999999,"5min":0.17999999999999999,"15min":0.16200000000000001},"max":{"1min":6.6280000000000001,"5min":14.365,"15min":14.365},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.98099999999999998,"15min":0.56000000000000005},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19700000000000001},"max":{"1min":6.6829999999999998,"5min":14.401,"15min":14.401},"last":0.27000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T10:01:45.330 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-13T10:01:45.330 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-13T10:01:45.598 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T10:01:45.598 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2024-09-13T10:01:45.610 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1598,"stamp":"2024-09-13T10:01:43.776627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":227,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4362244,"kb_used_data":15972,"kb_used_omap":4,"kb_used_meta":4346235,"kb_avail":278753276,"statfs":{"total":289910292480,"available":285443354624,"internally_reserved":0,"allocated":16355328,"data_stored":9122519,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":4450544978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536816694,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-117898,"num_read_kb":-39655415,"num_write":-625272,"num_write_kb":-231631040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-10,"num_bytes_recovered":-25165824,"num_keys_recovered":-4,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001164"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":474,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.305791+0000","last_change":"2024-09-13T09:23:52.448692+0000","last_active":"2024-09-13T10:01:39.305791+0000","last_peered":"2024-09-13T10:01:39.305791+0000","last_clean":"2024-09-13T10:01:39.305791+0000","last_became_active":"2024-09-13T09:23:50.442738+0000","last_became_peered":"2024-09-13T09:23:50.442738+0000","last_unstale":"2024-09-13T10:01:39.305791+0000","last_undegraded":"2024-09-13T10:01:39.305791+0000","last_fullsized":"2024-09-13T10:01:39.305791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:00:17.513367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072094900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":474,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.306055+0000","last_change":"2024-09-13T09:23:52.448736+0000","last_active":"2024-09-13T10:01:39.306055+0000","last_peered":"2024-09-13T10:01:39.306055+0000","last_clean":"2024-09-13T10:01:39.306055+0000","last_became_active":"2024-09-13T09:23:50.443108+0000","last_became_peered":"2024-09-13T09:23:50.443108+0000","last_unstale":"2024-09-13T10:01:39.306055+0000","last_undegraded":"2024-09-13T10:01:39.306055+0000","last_fullsized":"2024-09-13T10:01:39.306055+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:24.324564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064757699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":475,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.306453+0000","last_change":"2024-09-13T09:23:52.448503+0000","last_active":"2024-09-13T10:01:39.306453+0000","last_peered":"2024-09-13T10:01:39.306453+0000","last_clean":"2024-09-13T10:01:39.306453+0000","last_became_active":"2024-09-13T09:23:50.439969+0000","last_became_peered":"2024-09-13T09:23:50.439969+0000","last_unstale":"2024-09-13T10:01:39.306453+0000","last_undegraded":"2024-09-13T10:01:39.306453+0000","last_fullsized":"2024-09-13T10:01:39.306453+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:50.713752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052999299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":475,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.305027+0000","last_change":"2024-09-13T09:23:52.448963+0000","last_active":"2024-09-13T10:01:39.305027+0000","last_peered":"2024-09-13T10:01:39.305027+0000","last_clean":"2024-09-13T10:01:39.305027+0000","last_became_active":"2024-09-13T09:23:50.442812+0000","last_became_peered":"2024-09-13T09:23:50.442812+0000","last_unstale":"2024-09-13T10:01:39.305027+0000","last_undegraded":"2024-09-13T10:01:39.305027+0000","last_fullsized":"2024-09-13T10:01:39.305027+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:43.338946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088972200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":477,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.315618+0000","last_change":"2024-09-13T09:23:52.446617+0000","last_active":"2024-09-13T10:01:39.315618+0000","last_peered":"2024-09-13T10:01:39.315618+0000","last_clean":"2024-09-13T10:01:39.315618+0000","last_became_active":"2024-09-13T09:23:50.442111+0000","last_became_peered":"2024-09-13T09:23:50.442111+0000","last_unstale":"2024-09-13T10:01:39.315618+0000","last_undegraded":"2024-09-13T10:01:39.315618+0000","last_fullsized":"2024-09-13T10:01:39.315618+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:15.440943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000891642,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":474,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.309091+0000","last_change":"2024-09-13T09:23:52.443957+0000","last_active":"2024-09-13T10:01:39.309091+0000","last_peered":"2024-09-13T10:01:39.309091+0000","last_clean":"2024-09-13T10:01:39.309091+0000","last_became_active":"2024-09-13T09:23:50.441154+0000","last_became_peered":"2024-09-13T09:23:50.441154+0000","last_unstale":"2024-09-13T10:01:39.309091+0000","last_undegraded":"2024-09-13T10:01:39.309091+0000","last_fullsized":"2024-09-13T10:01:39.309091+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:40.913446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033978799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":474,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.310150+0000","last_change":"2024-09-13T09:23:52.444000+0000","last_active":"2024-09-13T10:01:39.310150+0000","last_peered":"2024-09-13T10:01:39.310150+0000","last_clean":"2024-09-13T10:01:39.310150+0000","last_became_active":"2024-09-13T09:23:50.441634+0000","last_became_peered":"2024-09-13T09:23:50.441634+0000","last_unstale":"2024-09-13T10:01:39.310150+0000","last_undegraded":"2024-09-13T10:01:39.310150+0000","last_fullsized":"2024-09-13T10:01:39.310150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:10.844086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038734700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":476,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.305125+0000","last_change":"2024-09-13T09:23:52.448440+0000","last_active":"2024-09-13T10:01:39.305125+0000","last_peered":"2024-09-13T10:01:39.305125+0000","last_clean":"2024-09-13T10:01:39.305125+0000","last_became_active":"2024-09-13T09:23:50.440112+0000","last_became_peered":"2024-09-13T09:23:50.440112+0000","last_unstale":"2024-09-13T10:01:39.305125+0000","last_undegraded":"2024-09-13T10:01:39.305125+0000","last_fullsized":"2024-09-13T10:01:39.305125+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:40.210218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048773600000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":600,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.304895+0000","last_change":"2024-09-13T09:23:45.887443+0000","last_active":"2024-09-13T10:01:39.304895+0000","last_peered":"2024-09-13T10:01:39.304895+0000","last_clean":"2024-09-13T10:01:39.304895+0000","last_became_active":"2024-09-13T09:23:45.886500+0000","last_became_peered":"2024-09-13T09:23:45.886500+0000","last_unstale":"2024-09-13T10:01:39.304895+0000","last_undegraded":"2024-09-13T10:01:39.304895+0000","last_fullsized":"2024-09-13T10:01:39.304895+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:44.266705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:44.266705+0000","last_clean_scrub_stamp":"2024-09-13T09:23:44.266705+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:28.855304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738831,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1300468,"kb_used_data":5288,"kb_used_omap":1,"kb_used_meta":1295166,"kb_avail":93071372,"statfs":{"total":96636764160,"available":95305084928,"internally_reserved":0,"allocated":5414912,"data_stored":2649626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1326250438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.69299999999999995,"15min":0.46500000000000002},"min":{"1min":0.182,"5min":0.14799999999999999,"15min":0.13700000000000001},"max":{"1min":7.9580000000000002,"5min":10.542,"15min":10.542},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.76400000000000001,"15min":0.53000000000000003},"min":{"1min":0.23599999999999999,"5min":0.19800000000000001,"15min":0.14299999999999999},"max":{"1min":7.9909999999999997,"5min":10.577999999999999,"15min":10.577999999999999},"last":0.40100000000000002}]},{"osd":1,"last update":"Fri Sep 13 10:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.48999999999999999,"15min":0.42499999999999999},"min":{"1min":0.19700000000000001,"5min":0.183,"15min":0.16},"max":{"1min":4.4210000000000003,"5min":4.4210000000000003,"15min":4.6429999999999998},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.115,"5min":0.85199999999999998,"15min":0.56299999999999994},"min":{"1min":0.219,"5min":0.193,"15min":0.16900000000000001},"max":{"1min":9.5009999999999994,"5min":9.8219999999999992,"15min":9.8219999999999992},"last":0.38600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738827,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1441504,"kb_used_data":5336,"kb_used_omap":1,"kb_used_meta":1436158,"kb_avail":92930336,"statfs":{"total":96636764160,"available":95160664064,"internally_reserved":0,"allocated":5464064,"data_stored":3232899,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1470626246},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:00:41 2024","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":0.84999999999999998,"15min":0.51500000000000001},"min":{"1min":0.23699999999999999,"5min":0.14299999999999999,"15min":0.13500000000000001},"max":{"1min":11.082000000000001,"5min":11.082000000000001,"15min":11.082000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63400000000000001,"15min":0.46700000000000003},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.17599999999999999},"max":{"1min":1.754,"5min":6.8369999999999997,"15min":6.9790000000000001},"last":0.35999999999999999}]},{"osd":2,"last update":"Fri Sep 13 10:00:41 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":0.73299999999999998,"15min":0.48899999999999999},"min":{"1min":0.20300000000000001,"5min":0.186,"15min":0.16900000000000001},"max":{"1min":11.034000000000001,"5min":11.034000000000001,"15min":11.034000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54800000000000004,"15min":0.40799999999999997},"min":{"1min":0.217,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.673,"5min":3.835,"15min":3.835},"last":0.312}]}]},{"osd":0,"up_from":8,"seq":34359738829,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1620272,"kb_used_data":5348,"kb_used_omap":1,"kb_used_meta":1614910,"kb_avail":92751568,"statfs":{"total":96636764160,"available":94977605632,"internally_reserved":0,"allocated":5476352,"data_stored":3239994,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1653668294},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 10:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.748,"15min":0.47299999999999998},"min":{"1min":0.19,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":7.5960000000000001,"5min":10.384,"15min":10.384},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.80800000000000005,"15min":0.52300000000000002},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.17799999999999999},"max":{"1min":7.6340000000000003,"5min":10.427,"15min":10.427},"last":0.25700000000000001}]},{"osd":2,"last update":"Fri Sep 13 10:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.91900000000000004,"15min":0.53400000000000003},"min":{"1min":0.20599999999999999,"5min":0.17999999999999999,"15min":0.16200000000000001},"max":{"1min":6.6280000000000001,"5min":14.365,"15min":14.365},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.98099999999999998,"15min":0.56000000000000005},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19700000000000001},"max":{"1min":6.6829999999999998,"5min":14.401,"15min":14.401},"last":0.27000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T10:01:45.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-13T10:01:45.881 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T10:01:45.881 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":459,"fsid":"4884b31c-ac19-4f07-b0a6-5fdd65ee73f4","created":"2024-09-13T09:23:33.956282+0000","modified":"2024-09-13T10:01:43.316490+0000","last_up_change":"2024-09-13T09:23:42.232423+0000","last_in_change":"2024-09-13T09:23:34.775774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T09:23:43.424493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T09:23:48.894975+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"49de89ee-dc0d-4ac6-a600-1530e8c29e9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":218678704},{"type":"v1","addr":"172.21.15.190:6801","nonce":218678704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":218678704},{"type":"v1","addr":"172.21.15.190:6803","nonce":218678704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":218678704},{"type":"v1","addr":"172.21.15.190:6807","nonce":218678704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":218678704},{"type":"v1","addr":"172.21.15.190:6805","nonce":218678704}]},"public_addr":"172.21.15.190:6801/218678704","cluster_addr":"172.21.15.190:6803/218678704","heartbeat_back_addr":"172.21.15.190:6807/218678704","heartbeat_front_addr":"172.21.15.190:6805/218678704","state":["exists","up"]},{"osd":1,"uuid":"143f28ae-3260-47f8-a94b-de76b062584d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6809","nonce":3841277131},{"type":"v1","addr":"172.21.15.190:6811","nonce":3841277131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6813","nonce":3841277131},{"type":"v1","addr":"172.21.15.190:6815","nonce":3841277131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6821","nonce":3841277131},{"type":"v1","addr":"172.21.15.190:6823","nonce":3841277131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6817","nonce":3841277131},{"type":"v1","addr":"172.21.15.190:6819","nonce":3841277131}]},"public_addr":"172.21.15.190:6811/3841277131","cluster_addr":"172.21.15.190:6815/3841277131","heartbeat_back_addr":"172.21.15.190:6823/3841277131","heartbeat_front_addr":"172.21.15.190:6819/3841277131","state":["exists","up"]},{"osd":2,"uuid":"19b0fabd-7a2b-4b10-a9da-b898d489df1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3245106308},{"type":"v1","addr":"172.21.15.190:6810","nonce":3245106308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3245106308},{"type":"v1","addr":"172.21.15.190:6814","nonce":3245106308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":3245106308},{"type":"v1","addr":"172.21.15.190:6822","nonce":3245106308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":3245106308},{"type":"v1","addr":"172.21.15.190:6818","nonce":3245106308}]},"public_addr":"172.21.15.190:6810/3245106308","cluster_addr":"172.21.15.190:6814/3245106308","heartbeat_back_addr":"172.21.15.190:6822/3245106308","heartbeat_front_addr":"172.21.15.190:6818/3245106308","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:39.855506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:39.931858+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:39.929265+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:0/1994336749":"2024-09-13T11:01:39.141234+0000","172.21.15.190:0/3635421876":"2024-09-13T10:59:39.342726+0000","172.21.15.190:0/1726999598":"2024-09-13T10:59:02.749758+0000","172.21.15.190:0/1175709403":"2024-09-13T10:58:20.905861+0000","172.21.15.190:0/1699714046":"2024-09-13T10:57:37.283257+0000","172.21.15.190:0/3229707289":"2024-09-13T10:56:52.318923+0000","172.21.15.190:0/1886263172":"2024-09-13T10:56:11.164376+0000","172.21.15.190:0/3145482867":"2024-09-13T10:52:49.514148+0000","172.21.15.190:0/1648057990":"2024-09-13T10:47:43.044151+0000","172.21.15.190:0/650857472":"2024-09-13T10:51:46.065325+0000","172.21.15.190:0/3526939122":"2024-09-13T10:45:00.431949+0000","172.21.15.190:0/2725636392":"2024-09-13T10:46:24.959522+0000","172.21.15.190:0/3177020632":"2024-09-13T10:50:16.063793+0000","172.21.15.190:0/3847791812":"2024-09-13T10:46:07.145577+0000","172.21.15.190:0/2104190865":"2024-09-13T10:45:41.873337+0000","172.21.15.190:0/1331679647":"2024-09-13T10:45:58.161303+0000","172.21.15.190:0/3924021838":"2024-09-13T10:51:37.658278+0000","172.21.15.190:0/2863964976":"2024-09-13T10:47:13.422703+0000","172.21.15.190:0/358316942":"2024-09-13T11:00:58.874371+0000","172.21.15.190:0/556673905":"2024-09-13T10:45:49.907126+0000","172.21.15.190:0/4159116097":"2024-09-13T10:55:29.117205+0000","172.21.15.190:0/2156538648":"2024-09-13T10:47:06.402331+0000","172.21.15.190:0/710769303":"2024-09-13T10:49:53.876658+0000","172.21.15.190:0/158890654":"2024-09-13T10:31:47.515316+0000","172.21.15.190:0/2680825658":"2024-09-13T10:26:53.367637+0000","172.21.15.190:0/2959131903":"2024-09-13T11:00:21.894713+0000","172.21.15.190:0/3835585015":"2024-09-13T10:27:48.852111+0000","172.21.15.190:0/26561117":"2024-09-13T10:46:38.307942+0000","172.21.15.190:0/3823892752":"2024-09-13T10:49:04.015169+0000","172.21.15.190:0/1033823955":"2024-09-13T10:28:35.848265+0000","172.21.15.190:0/1461075616":"2024-09-13T10:29:30.938231+0000","172.21.15.190:0/784938607":"2024-09-13T10:46:16.147830+0000","172.21.15.190:0/2182744793":"2024-09-13T10:46:59.408574+0000","172.21.15.190:0/2343791881":"2024-09-13T10:34:34.884914+0000","172.21.15.190:0/3189416350":"2024-09-13T10:45:33.026466+0000","172.21.15.190:0/3360806828":"2024-09-13T10:46:52.369786+0000","172.21.15.190:0/978674951":"2024-09-13T10:32:43.150653+0000","172.21.15.190:0/66499452":"2024-09-13T10:48:55.570468+0000","172.21.15.190:0/3795419274":"2024-09-13T10:47:27.842679+0000","172.21.15.190:0/3436440327":"2024-09-13T10:50:09.593602+0000","172.21.15.190:0/2599820642":"2024-09-13T10:26:08.065695+0000","172.21.15.190:0/3399290791":"2024-09-13T10:33:40.962079+0000","172.21.15.190:0/1136678078":"2024-09-13T10:51:29.649937+0000","172.21.15.190:0/1486734927":"2024-09-13T10:52:43.133111+0000","172.21.15.190:0/2356363365":"2024-09-13T10:45:08.839688+0000","172.21.15.190:0/785390366":"2024-09-13T10:47:49.578755+0000","172.21.15.190:0/2148159319":"2024-09-13T10:48:47.550879+0000","172.21.15.190:0/1684612142":"2024-09-13T10:46:31.398969+0000","172.21.15.190:0/2203584916":"2024-09-13T10:47:34.839813+0000","172.21.15.190:0/320571934":"2024-09-13T10:46:45.336897+0000","172.21.15.190:0/4165559035":"2024-09-13T10:47:58.079738+0000","172.21.15.190:0/2663494650":"2024-09-13T10:51:20.899636+0000","172.21.15.190:0/2122604777":"2024-09-13T10:48:06.123080+0000","172.21.15.190:0/2557078863":"2024-09-13T10:49:46.726340+0000","172.21.15.190:0/3384348507":"2024-09-13T10:30:26.405904+0000","172.21.15.190:0/894686178":"2024-09-13T10:48:14.114237+0000","172.21.15.190:0/4117275049":"2024-09-13T10:48:20.887993+0000","172.21.15.190:0/2638216932":"2024-09-13T10:51:54.071278+0000","172.21.15.190:0/4097123634":"2024-09-13T10:48:28.045062+0000","172.21.15.190:0/227918356":"2024-09-13T10:48:38.886528+0000","172.21.15.190:0/2437209183":"2024-09-13T10:49:17.403155+0000","172.21.15.190:0/1447185498":"2024-09-13T10:49:24.417295+0000","172.21.15.190:0/1548568086":"2024-09-13T10:49:10.354305+0000","172.21.15.190:0/2779061428":"2024-09-13T10:52:35.146893+0000","172.21.15.190:0/2728944946":"2024-09-13T10:49:31.415988+0000","172.21.15.190:0/1718748896":"2024-09-13T10:50:22.913342+0000","172.21.15.190:0/3080583148":"2024-09-13T10:49:38.501963+0000","172.21.15.190:0/2663443288":"2024-09-13T10:50:00.868155+0000","172.21.15.190:0/1555184777":"2024-09-13T10:50:31.591585+0000","172.21.15.190:0/1604184506":"2024-09-13T10:51:12.619155+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T10:01:46.894 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-09-13T10:01:46.895 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-13T10:01:47.027 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-13T10:01:47.027 INFO:teuthology.orchestra.run.smithi190.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T10:01:47.027 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-13T10:01:47.038 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-09-13T10:01:47.307 INFO:teuthology.orchestra.run.smithi190.stderr:instructed osd(s) 0 to deep-scrub 2024-09-13T10:01:47.319 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-09-13T10:01:47.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-13T10:01:47.439 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-13T10:01:47.439 INFO:teuthology.orchestra.run.smithi190.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T10:01:47.439 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-13T10:01:47.449 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-09-13T10:01:47.719 INFO:teuthology.orchestra.run.smithi190.stderr:instructed osd(s) 1 to deep-scrub 2024-09-13T10:01:47.732 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-09-13T10:01:47.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-13T10:01:47.853 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-13T10:01:47.853 INFO:teuthology.orchestra.run.smithi190.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T10:01:47.853 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-13T10:01:47.863 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-09-13T10:01:48.131 INFO:teuthology.orchestra.run.smithi190.stderr:instructed osd(s) 2 to deep-scrub 2024-09-13T10:01:48.144 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-13T10:01:48.413 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T10:01:48.413 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2024-09-13T10:01:48.422 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1600,"stamp":"2024-09-13T10:01:47.777196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4362276,"kb_used_data":16000,"kb_used_omap":4,"kb_used_meta":4346235,"kb_avail":278753244,"statfs":{"total":289910292480,"available":285443321856,"internally_reserved":0,"allocated":16384000,"data_stored":9136254,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":4450544978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536816694,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-117909,"num_read_kb":-39655425,"num_write":-625273,"num_write_kb":-231631041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-10,"num_bytes_recovered":-25165824,"num_keys_recovered":-4,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001202"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":475,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-09-13T10:01:41.880266+0000","last_change":"2024-09-13T09:23:52.448692+0000","last_active":"2024-09-13T10:01:41.880266+0000","last_peered":"2024-09-13T10:01:41.880266+0000","last_clean":"2024-09-13T10:01:41.880266+0000","last_became_active":"2024-09-13T09:23:50.442738+0000","last_became_peered":"2024-09-13T09:23:50.442738+0000","last_unstale":"2024-09-13T10:01:41.880266+0000","last_undegraded":"2024-09-13T10:01:41.880266+0000","last_fullsized":"2024-09-13T10:01:41.880266+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:00:17.513367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072094900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":475,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-09-13T10:01:41.887042+0000","last_change":"2024-09-13T09:23:52.448736+0000","last_active":"2024-09-13T10:01:41.887042+0000","last_peered":"2024-09-13T10:01:41.887042+0000","last_clean":"2024-09-13T10:01:41.887042+0000","last_became_active":"2024-09-13T09:23:50.443108+0000","last_became_peered":"2024-09-13T09:23:50.443108+0000","last_unstale":"2024-09-13T10:01:41.887042+0000","last_undegraded":"2024-09-13T10:01:41.887042+0000","last_fullsized":"2024-09-13T10:01:41.887042+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:24.324564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064757699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":476,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-09-13T10:01:41.882416+0000","last_change":"2024-09-13T09:23:52.448503+0000","last_active":"2024-09-13T10:01:41.882416+0000","last_peered":"2024-09-13T10:01:41.882416+0000","last_clean":"2024-09-13T10:01:41.882416+0000","last_became_active":"2024-09-13T09:23:50.439969+0000","last_became_peered":"2024-09-13T09:23:50.439969+0000","last_unstale":"2024-09-13T10:01:41.882416+0000","last_undegraded":"2024-09-13T10:01:41.882416+0000","last_fullsized":"2024-09-13T10:01:41.882416+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:50.713752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052999299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":476,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-09-13T10:01:41.881649+0000","last_change":"2024-09-13T09:23:52.448963+0000","last_active":"2024-09-13T10:01:41.881649+0000","last_peered":"2024-09-13T10:01:41.881649+0000","last_clean":"2024-09-13T10:01:41.881649+0000","last_became_active":"2024-09-13T09:23:50.442812+0000","last_became_peered":"2024-09-13T09:23:50.442812+0000","last_unstale":"2024-09-13T10:01:41.881649+0000","last_undegraded":"2024-09-13T10:01:41.881649+0000","last_fullsized":"2024-09-13T10:01:41.881649+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:43.338946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088972200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":477,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.315618+0000","last_change":"2024-09-13T09:23:52.446617+0000","last_active":"2024-09-13T10:01:39.315618+0000","last_peered":"2024-09-13T10:01:39.315618+0000","last_clean":"2024-09-13T10:01:39.315618+0000","last_became_active":"2024-09-13T09:23:50.442111+0000","last_became_peered":"2024-09-13T09:23:50.442111+0000","last_unstale":"2024-09-13T10:01:39.315618+0000","last_undegraded":"2024-09-13T10:01:39.315618+0000","last_fullsized":"2024-09-13T10:01:39.315618+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:15.440943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000891642,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":474,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.309091+0000","last_change":"2024-09-13T09:23:52.443957+0000","last_active":"2024-09-13T10:01:39.309091+0000","last_peered":"2024-09-13T10:01:39.309091+0000","last_clean":"2024-09-13T10:01:39.309091+0000","last_became_active":"2024-09-13T09:23:50.441154+0000","last_became_peered":"2024-09-13T09:23:50.441154+0000","last_unstale":"2024-09-13T10:01:39.309091+0000","last_undegraded":"2024-09-13T10:01:39.309091+0000","last_fullsized":"2024-09-13T10:01:39.309091+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:40.913446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033978799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":474,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-09-13T10:01:39.310150+0000","last_change":"2024-09-13T09:23:52.444000+0000","last_active":"2024-09-13T10:01:39.310150+0000","last_peered":"2024-09-13T10:01:39.310150+0000","last_clean":"2024-09-13T10:01:39.310150+0000","last_became_active":"2024-09-13T09:23:50.441634+0000","last_became_peered":"2024-09-13T09:23:50.441634+0000","last_unstale":"2024-09-13T10:01:39.310150+0000","last_undegraded":"2024-09-13T10:01:39.310150+0000","last_fullsized":"2024-09-13T10:01:39.310150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:10.844086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038734700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":477,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-09-13T10:01:41.881478+0000","last_change":"2024-09-13T09:23:52.448440+0000","last_active":"2024-09-13T10:01:41.881478+0000","last_peered":"2024-09-13T10:01:41.881478+0000","last_clean":"2024-09-13T10:01:41.881478+0000","last_became_active":"2024-09-13T09:23:50.440112+0000","last_became_peered":"2024-09-13T09:23:50.440112+0000","last_unstale":"2024-09-13T10:01:41.881478+0000","last_undegraded":"2024-09-13T10:01:41.881478+0000","last_fullsized":"2024-09-13T10:01:41.881478+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:40.210218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048773600000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":601,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-09-13T10:01:41.880068+0000","last_change":"2024-09-13T09:23:45.887443+0000","last_active":"2024-09-13T10:01:41.880068+0000","last_peered":"2024-09-13T10:01:41.880068+0000","last_clean":"2024-09-13T10:01:41.880068+0000","last_became_active":"2024-09-13T09:23:45.886500+0000","last_became_peered":"2024-09-13T09:23:45.886500+0000","last_unstale":"2024-09-13T10:01:41.880068+0000","last_undegraded":"2024-09-13T10:01:41.880068+0000","last_fullsized":"2024-09-13T10:01:41.880068+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:44.266705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:44.266705+0000","last_clean_scrub_stamp":"2024-09-13T09:23:44.266705+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:28.855304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738831,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1300468,"kb_used_data":5288,"kb_used_omap":1,"kb_used_meta":1295166,"kb_avail":93071372,"statfs":{"total":96636764160,"available":95305084928,"internally_reserved":0,"allocated":5414912,"data_stored":2649626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1326250438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.69299999999999995,"15min":0.46500000000000002},"min":{"1min":0.182,"5min":0.14799999999999999,"15min":0.13700000000000001},"max":{"1min":7.9580000000000002,"5min":10.542,"15min":10.542},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.76400000000000001,"15min":0.53000000000000003},"min":{"1min":0.23599999999999999,"5min":0.19800000000000001,"15min":0.14299999999999999},"max":{"1min":7.9909999999999997,"5min":10.577999999999999,"15min":10.577999999999999},"last":0.40100000000000002}]},{"osd":1,"last update":"Fri Sep 13 10:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.48999999999999999,"15min":0.42499999999999999},"min":{"1min":0.19700000000000001,"5min":0.183,"15min":0.16},"max":{"1min":4.4210000000000003,"5min":4.4210000000000003,"15min":4.6429999999999998},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.115,"5min":0.85199999999999998,"15min":0.56299999999999994},"min":{"1min":0.219,"5min":0.193,"15min":0.16900000000000001},"max":{"1min":9.5009999999999994,"5min":9.8219999999999992,"15min":9.8219999999999992},"last":0.38600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738828,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1441536,"kb_used_data":5364,"kb_used_omap":1,"kb_used_meta":1436158,"kb_avail":92930304,"statfs":{"total":96636764160,"available":95160631296,"internally_reserved":0,"allocated":5492736,"data_stored":3246634,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1470626246},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:01:43 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.71299999999999997,"15min":0.51700000000000002},"min":{"1min":0.214,"5min":0.14299999999999999,"15min":0.13500000000000001},"max":{"1min":1.0449999999999999,"5min":11.082000000000001,"15min":11.082000000000001},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.371,"5min":0.53600000000000003,"15min":0.46800000000000003},"min":{"1min":0.25,"5min":0.20399999999999999,"15min":0.17599999999999999},"max":{"1min":1.0229999999999999,"5min":5.6059999999999999,"15min":6.9790000000000001},"last":0.28299999999999997}]},{"osd":2,"last update":"Fri Sep 13 10:01:43 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.66800000000000004,"15min":0.48999999999999999},"min":{"1min":0.23400000000000001,"5min":0.186,"15min":0.16900000000000001},"max":{"1min":1.1259999999999999,"5min":11.034000000000001,"15min":11.034000000000001},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.30099999999999999,"5min":0.45500000000000002,"15min":0.40699999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.44,"5min":3.835,"15min":3.835},"last":0.22800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738829,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1620272,"kb_used_data":5348,"kb_used_omap":1,"kb_used_meta":1614910,"kb_avail":92751568,"statfs":{"total":96636764160,"available":94977605632,"internally_reserved":0,"allocated":5476352,"data_stored":3239994,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1653668294},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 10:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.748,"15min":0.47299999999999998},"min":{"1min":0.19,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":7.5960000000000001,"5min":10.384,"15min":10.384},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.80800000000000005,"15min":0.52300000000000002},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.17799999999999999},"max":{"1min":7.6340000000000003,"5min":10.427,"15min":10.427},"last":0.25700000000000001}]},{"osd":2,"last update":"Fri Sep 13 10:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.91900000000000004,"15min":0.53400000000000003},"min":{"1min":0.20599999999999999,"5min":0.17999999999999999,"15min":0.16200000000000001},"max":{"1min":6.6280000000000001,"5min":14.365,"15min":14.365},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.98099999999999998,"15min":0.56000000000000005},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19700000000000001},"max":{"1min":6.6829999999999998,"5min":14.401,"15min":14.401},"last":0.27000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T10:01:48.426 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-13T09:23:49.423953+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=9, tm_min=23, tm_sec=49, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=10, tm_min=1, tm_sec=45, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T10:01:48.427 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-13T09:23:49.423953+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=9, tm_min=23, tm_sec=49, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=10, tm_min=1, tm_sec=45, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T10:01:48.427 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-13T09:23:49.423953+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=9, tm_min=23, tm_sec=49, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=10, tm_min=1, tm_sec=45, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T10:01:48.427 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-13T09:23:49.423953+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=9, tm_min=23, tm_sec=49, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=10, tm_min=1, tm_sec=45, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T10:01:48.427 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-13T09:23:49.423953+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=9, tm_min=23, tm_sec=49, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=10, tm_min=1, tm_sec=45, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T10:01:48.427 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-13T09:23:49.423953+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=9, tm_min=23, tm_sec=49, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=10, tm_min=1, tm_sec=45, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T10:01:48.427 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-13T09:23:49.423953+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=9, tm_min=23, tm_sec=49, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=10, tm_min=1, tm_sec=45, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T10:01:48.427 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-13T09:23:49.423953+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=9, tm_min=23, tm_sec=49, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=10, tm_min=1, tm_sec=45, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T10:01:48.427 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-13T09:23:44.266705+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=9, tm_min=23, tm_sec=44, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=10, tm_min=1, tm_sec=45, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T10:01:48.428 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-13T10:02:08.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-13T10:02:08.710 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T10:02:08.710 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2024-09-13T10:02:08.722 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1610,"stamp":"2024-09-13T10:02:07.781116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4362292,"kb_used_data":16028,"kb_used_omap":4,"kb_used_meta":4346235,"kb_avail":278753228,"statfs":{"total":289910292480,"available":285443305472,"internally_reserved":0,"allocated":16412672,"data_stored":9149515,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":4450544978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002202"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":487,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-13T10:02:04.934153+0000","last_change":"2024-09-13T10:02:04.934275+0000","last_active":"2024-09-13T10:02:04.934153+0000","last_peered":"2024-09-13T10:02:04.934153+0000","last_clean":"2024-09-13T10:02:04.934153+0000","last_became_active":"2024-09-13T09:23:50.442738+0000","last_became_peered":"2024-09-13T09:23:50.442738+0000","last_unstale":"2024-09-13T10:02:04.934153+0000","last_undegraded":"2024-09-13T10:02:04.934153+0000","last_fullsized":"2024-09-13T10:02:04.934153+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:02:04.934131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:02:04.934131+0000","last_clean_scrub_stamp":"2024-09-13T10:02:04.934131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:05:01.176783+0000","scrub_duration":0.0015869600000000001,"objects_trimmed":0,"snaptrim_duration":0.00072094900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":487,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-13T10:02:01.909181+0000","last_change":"2024-09-13T10:02:01.909294+0000","last_active":"2024-09-13T10:02:01.909181+0000","last_peered":"2024-09-13T10:02:01.909181+0000","last_clean":"2024-09-13T10:02:01.909181+0000","last_became_active":"2024-09-13T09:23:50.443108+0000","last_became_peered":"2024-09-13T09:23:50.443108+0000","last_unstale":"2024-09-13T10:02:01.909181+0000","last_undegraded":"2024-09-13T10:02:01.909181+0000","last_fullsized":"2024-09-13T10:02:01.909181+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:02:01.909161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:02:01.909161+0000","last_clean_scrub_stamp":"2024-09-13T10:02:01.909161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:55.703533+0000","scrub_duration":0.001506447,"objects_trimmed":0,"snaptrim_duration":0.00064757699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":477,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-13T10:01:44.334837+0000","last_change":"2024-09-13T09:23:52.448503+0000","last_active":"2024-09-13T10:01:44.334837+0000","last_peered":"2024-09-13T10:01:44.334837+0000","last_clean":"2024-09-13T10:01:44.334837+0000","last_became_active":"2024-09-13T09:23:50.439969+0000","last_became_peered":"2024-09-13T09:23:50.439969+0000","last_unstale":"2024-09-13T10:01:44.334837+0000","last_undegraded":"2024-09-13T10:01:44.334837+0000","last_fullsized":"2024-09-13T10:01:44.334837+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:49.423953+0000","last_clean_scrub_stamp":"2024-09-13T09:23:49.423953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052999299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":488,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-13T10:01:57.988748+0000","last_change":"2024-09-13T10:01:57.988859+0000","last_active":"2024-09-13T10:01:57.988748+0000","last_peered":"2024-09-13T10:01:57.988748+0000","last_clean":"2024-09-13T10:01:57.988748+0000","last_became_active":"2024-09-13T09:23:50.442812+0000","last_became_peered":"2024-09-13T09:23:50.442812+0000","last_unstale":"2024-09-13T10:01:57.988748+0000","last_undegraded":"2024-09-13T10:01:57.988748+0000","last_fullsized":"2024-09-13T10:01:57.988748+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:01:57.988728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:01:57.988728+0000","last_clean_scrub_stamp":"2024-09-13T10:01:57.988728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:56:57.701679+0000","scrub_duration":0.001554492,"objects_trimmed":0,"snaptrim_duration":0.00088972200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"459'3","reported_seq":491,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-13T10:01:53.541471+0000","last_change":"2024-09-13T10:01:53.541571+0000","last_active":"2024-09-13T10:01:53.541471+0000","last_peered":"2024-09-13T10:01:53.541471+0000","last_clean":"2024-09-13T10:01:53.541471+0000","last_became_active":"2024-09-13T09:23:50.442111+0000","last_became_peered":"2024-09-13T09:23:50.442111+0000","last_unstale":"2024-09-13T10:01:53.541471+0000","last_undegraded":"2024-09-13T10:01:53.541471+0000","last_fullsized":"2024-09-13T10:01:53.541471+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'3","last_scrub_stamp":"2024-09-13T10:01:53.541448+0000","last_deep_scrub":"459'3","last_deep_scrub_stamp":"2024-09-13T10:01:53.541448+0000","last_clean_scrub_stamp":"2024-09-13T10:01:53.541448+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:04.206949+0000","scrub_duration":0.002704382,"objects_trimmed":0,"snaptrim_duration":0.000891642,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":487,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-13T10:01:52.942584+0000","last_change":"2024-09-13T10:01:52.942715+0000","last_active":"2024-09-13T10:01:52.942584+0000","last_peered":"2024-09-13T10:01:52.942584+0000","last_clean":"2024-09-13T10:01:52.942584+0000","last_became_active":"2024-09-13T09:23:50.441154+0000","last_became_peered":"2024-09-13T09:23:50.441154+0000","last_unstale":"2024-09-13T10:01:52.942584+0000","last_undegraded":"2024-09-13T10:01:52.942584+0000","last_fullsized":"2024-09-13T10:01:52.942584+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:01:52.942551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:01:52.942551+0000","last_clean_scrub_stamp":"2024-09-13T10:01:52.942551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:15:52.487389+0000","scrub_duration":0.001556672,"objects_trimmed":0,"snaptrim_duration":0.00033978799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":488,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-13T10:01:49.922196+0000","last_change":"2024-09-13T10:01:49.922196+0000","last_active":"2024-09-13T10:01:49.922196+0000","last_peered":"2024-09-13T10:01:49.922196+0000","last_clean":"2024-09-13T10:01:49.922196+0000","last_became_active":"2024-09-13T09:23:50.441634+0000","last_became_peered":"2024-09-13T09:23:50.441634+0000","last_unstale":"2024-09-13T10:01:49.922196+0000","last_undegraded":"2024-09-13T10:01:49.922196+0000","last_fullsized":"2024-09-13T10:01:49.922196+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:01:49.922016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:01:49.922016+0000","last_clean_scrub_stamp":"2024-09-13T10:01:49.922016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:57.799098+0000","scrub_duration":0.0015797980000000001,"objects_trimmed":0,"snaptrim_duration":0.00038734700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"459'2","reported_seq":490,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-13T10:01:59.945752+0000","last_change":"2024-09-13T10:01:59.945866+0000","last_active":"2024-09-13T10:01:59.945752+0000","last_peered":"2024-09-13T10:01:59.945752+0000","last_clean":"2024-09-13T10:01:59.945752+0000","last_became_active":"2024-09-13T09:23:50.440112+0000","last_became_peered":"2024-09-13T09:23:50.440112+0000","last_unstale":"2024-09-13T10:01:59.945752+0000","last_undegraded":"2024-09-13T10:01:59.945752+0000","last_fullsized":"2024-09-13T10:01:59.945752+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'2","last_scrub_stamp":"2024-09-13T10:01:59.945729+0000","last_deep_scrub":"459'2","last_deep_scrub_stamp":"2024-09-13T10:01:59.945729+0000","last_clean_scrub_stamp":"2024-09-13T10:01:59.945729+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:34.724567+0000","scrub_duration":0.002812443,"objects_trimmed":0,"snaptrim_duration":0.00048773600000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"459'99","reported_seq":616,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-13T10:01:48.919365+0000","last_change":"2024-09-13T10:01:48.919365+0000","last_active":"2024-09-13T10:01:48.919365+0000","last_peered":"2024-09-13T10:01:48.919365+0000","last_clean":"2024-09-13T10:01:48.919365+0000","last_became_active":"2024-09-13T09:23:45.886500+0000","last_became_peered":"2024-09-13T09:23:45.886500+0000","last_unstale":"2024-09-13T10:01:48.919365+0000","last_undegraded":"2024-09-13T10:01:48.919365+0000","last_fullsized":"2024-09-13T10:01:48.919365+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'99","last_scrub_stamp":"2024-09-13T10:01:48.919247+0000","last_deep_scrub":"459'99","last_deep_scrub_stamp":"2024-09-13T10:01:48.919247+0000","last_clean_scrub_stamp":"2024-09-13T10:01:48.919247+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:50:18.539681+0000","scrub_duration":0.0036986100000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738835,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1300476,"kb_used_data":5300,"kb_used_omap":1,"kb_used_meta":1295166,"kb_avail":93071364,"statfs":{"total":96636764160,"available":95305076736,"internally_reserved":0,"allocated":5427200,"data_stored":2656247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1326250438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:01:53 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.56200000000000006,"15min":0.46800000000000003},"min":{"1min":0.19500000000000001,"5min":0.159,"15min":0.14099999999999999},"max":{"1min":0.69199999999999995,"5min":7.9580000000000002,"15min":10.542},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.63400000000000001,"15min":0.53400000000000003},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.14299999999999999},"max":{"1min":0.71599999999999997,"5min":7.9909999999999997,"15min":10.577999999999999},"last":0.372}]},{"osd":1,"last update":"Fri Sep 13 10:01:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47799999999999998,"15min":0.437},"min":{"1min":0.26900000000000002,"5min":0.183,"15min":0.16500000000000001},"max":{"1min":2.6160000000000001,"5min":4.4210000000000003,"15min":4.6429999999999998},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.70999999999999996,"15min":0.57999999999999996},"min":{"1min":0.215,"5min":0.215,"15min":0.16900000000000001},"max":{"1min":2.4860000000000002,"5min":9.5009999999999994,"15min":9.8219999999999992},"last":0.308}]}]},{"osd":1,"up_from":8,"seq":34359738832,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1441532,"kb_used_data":5364,"kb_used_omap":1,"kb_used_meta":1436158,"kb_avail":92930308,"statfs":{"total":96636764160,"available":95160635392,"internally_reserved":0,"allocated":5492736,"data_stored":3246634,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1470626246},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:01:43 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.71299999999999997,"15min":0.51700000000000002},"min":{"1min":0.214,"5min":0.14299999999999999,"15min":0.13500000000000001},"max":{"1min":1.0449999999999999,"5min":11.082000000000001,"15min":11.082000000000001},"last":0.436},{"interface":"front","average":{"1min":0.371,"5min":0.53600000000000003,"15min":0.46800000000000003},"min":{"1min":0.25,"5min":0.20399999999999999,"15min":0.17599999999999999},"max":{"1min":1.0229999999999999,"5min":5.6059999999999999,"15min":6.9790000000000001},"last":0.45700000000000002}]},{"osd":2,"last update":"Fri Sep 13 10:01:43 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.66800000000000004,"15min":0.48999999999999999},"min":{"1min":0.23400000000000001,"5min":0.186,"15min":0.16900000000000001},"max":{"1min":1.1259999999999999,"5min":11.034000000000001,"15min":11.034000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.30099999999999999,"5min":0.45500000000000002,"15min":0.40699999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.44,"5min":3.835,"15min":3.835},"last":0.42199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738833,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1620284,"kb_used_data":5364,"kb_used_omap":1,"kb_used_meta":1614910,"kb_avail":92751556,"statfs":{"total":96636764160,"available":94977593344,"internally_reserved":0,"allocated":5492736,"data_stored":3246634,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1653668294},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 10:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.748,"15min":0.47299999999999998},"min":{"1min":0.19,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":7.5960000000000001,"5min":10.384,"15min":10.384},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.80800000000000005,"15min":0.52300000000000002},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.17799999999999999},"max":{"1min":7.6340000000000003,"5min":10.427,"15min":10.427},"last":0.40400000000000003}]},{"osd":2,"last update":"Fri Sep 13 10:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.91900000000000004,"15min":0.53400000000000003},"min":{"1min":0.20599999999999999,"5min":0.17999999999999999,"15min":0.16200000000000001},"max":{"1min":6.6280000000000001,"5min":14.365,"15min":14.365},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.98099999999999998,"15min":0.56000000000000005},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.19700000000000001},"max":{"1min":6.6829999999999998,"5min":14.401,"15min":14.401},"last":0.45600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T10:02:08.724 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-13T09:23:49.423953+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=9, tm_min=23, tm_sec=49, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=10, tm_min=1, tm_sec=45, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T10:02:08.724 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-13T10:02:28.725 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-13T10:02:29.007 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-13T10:02:29.007 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2024-09-13T10:02:29.019 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":1620,"stamp":"2024-09-13T10:02:27.785163+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4362292,"kb_used_data":16028,"kb_used_omap":4,"kb_used_meta":4346235,"kb_avail":278753228,"statfs":{"total":289910292480,"available":285443305472,"internally_reserved":0,"allocated":16412672,"data_stored":9149515,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":4450544978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002531"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":487,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-13T10:02:04.934153+0000","last_change":"2024-09-13T10:02:04.934275+0000","last_active":"2024-09-13T10:02:04.934153+0000","last_peered":"2024-09-13T10:02:04.934153+0000","last_clean":"2024-09-13T10:02:04.934153+0000","last_became_active":"2024-09-13T09:23:50.442738+0000","last_became_peered":"2024-09-13T09:23:50.442738+0000","last_unstale":"2024-09-13T10:02:04.934153+0000","last_undegraded":"2024-09-13T10:02:04.934153+0000","last_fullsized":"2024-09-13T10:02:04.934153+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:02:04.934131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:02:04.934131+0000","last_clean_scrub_stamp":"2024-09-13T10:02:04.934131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:05:01.176783+0000","scrub_duration":0.0015869600000000001,"objects_trimmed":0,"snaptrim_duration":0.00072094900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":487,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-13T10:02:01.909181+0000","last_change":"2024-09-13T10:02:01.909294+0000","last_active":"2024-09-13T10:02:01.909181+0000","last_peered":"2024-09-13T10:02:01.909181+0000","last_clean":"2024-09-13T10:02:01.909181+0000","last_became_active":"2024-09-13T09:23:50.443108+0000","last_became_peered":"2024-09-13T09:23:50.443108+0000","last_unstale":"2024-09-13T10:02:01.909181+0000","last_undegraded":"2024-09-13T10:02:01.909181+0000","last_fullsized":"2024-09-13T10:02:01.909181+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:02:01.909161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:02:01.909161+0000","last_clean_scrub_stamp":"2024-09-13T10:02:01.909161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:55.703533+0000","scrub_duration":0.001506447,"objects_trimmed":0,"snaptrim_duration":0.00064757699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":488,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-13T10:02:07.931592+0000","last_change":"2024-09-13T10:02:07.931703+0000","last_active":"2024-09-13T10:02:07.931592+0000","last_peered":"2024-09-13T10:02:07.931592+0000","last_clean":"2024-09-13T10:02:07.931592+0000","last_became_active":"2024-09-13T09:23:50.439969+0000","last_became_peered":"2024-09-13T09:23:50.439969+0000","last_unstale":"2024-09-13T10:02:07.931592+0000","last_undegraded":"2024-09-13T10:02:07.931592+0000","last_fullsized":"2024-09-13T10:02:07.931592+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:02:07.931572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:02:07.931572+0000","last_clean_scrub_stamp":"2024-09-13T10:02:07.931572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:15:43.145921+0000","scrub_duration":0.001564275,"objects_trimmed":0,"snaptrim_duration":0.00052999299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":488,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-13T10:01:57.988748+0000","last_change":"2024-09-13T10:01:57.988859+0000","last_active":"2024-09-13T10:01:57.988748+0000","last_peered":"2024-09-13T10:01:57.988748+0000","last_clean":"2024-09-13T10:01:57.988748+0000","last_became_active":"2024-09-13T09:23:50.442812+0000","last_became_peered":"2024-09-13T09:23:50.442812+0000","last_unstale":"2024-09-13T10:01:57.988748+0000","last_undegraded":"2024-09-13T10:01:57.988748+0000","last_fullsized":"2024-09-13T10:01:57.988748+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:01:57.988728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:01:57.988728+0000","last_clean_scrub_stamp":"2024-09-13T10:01:57.988728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:56:57.701679+0000","scrub_duration":0.001554492,"objects_trimmed":0,"snaptrim_duration":0.00088972200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"459'3","reported_seq":491,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-13T10:01:53.541471+0000","last_change":"2024-09-13T10:01:53.541571+0000","last_active":"2024-09-13T10:01:53.541471+0000","last_peered":"2024-09-13T10:01:53.541471+0000","last_clean":"2024-09-13T10:01:53.541471+0000","last_became_active":"2024-09-13T09:23:50.442111+0000","last_became_peered":"2024-09-13T09:23:50.442111+0000","last_unstale":"2024-09-13T10:01:53.541471+0000","last_undegraded":"2024-09-13T10:01:53.541471+0000","last_fullsized":"2024-09-13T10:01:53.541471+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'3","last_scrub_stamp":"2024-09-13T10:01:53.541448+0000","last_deep_scrub":"459'3","last_deep_scrub_stamp":"2024-09-13T10:01:53.541448+0000","last_clean_scrub_stamp":"2024-09-13T10:01:53.541448+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:04.206949+0000","scrub_duration":0.002704382,"objects_trimmed":0,"snaptrim_duration":0.000891642,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":487,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-13T10:01:52.942584+0000","last_change":"2024-09-13T10:01:52.942715+0000","last_active":"2024-09-13T10:01:52.942584+0000","last_peered":"2024-09-13T10:01:52.942584+0000","last_clean":"2024-09-13T10:01:52.942584+0000","last_became_active":"2024-09-13T09:23:50.441154+0000","last_became_peered":"2024-09-13T09:23:50.441154+0000","last_unstale":"2024-09-13T10:01:52.942584+0000","last_undegraded":"2024-09-13T10:01:52.942584+0000","last_fullsized":"2024-09-13T10:01:52.942584+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:01:52.942551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:01:52.942551+0000","last_clean_scrub_stamp":"2024-09-13T10:01:52.942551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:15:52.487389+0000","scrub_duration":0.001556672,"objects_trimmed":0,"snaptrim_duration":0.00033978799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":488,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-13T10:01:49.922196+0000","last_change":"2024-09-13T10:01:49.922196+0000","last_active":"2024-09-13T10:01:49.922196+0000","last_peered":"2024-09-13T10:01:49.922196+0000","last_clean":"2024-09-13T10:01:49.922196+0000","last_became_active":"2024-09-13T09:23:50.441634+0000","last_became_peered":"2024-09-13T09:23:50.441634+0000","last_unstale":"2024-09-13T10:01:49.922196+0000","last_undegraded":"2024-09-13T10:01:49.922196+0000","last_fullsized":"2024-09-13T10:01:49.922196+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:01:49.922016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:01:49.922016+0000","last_clean_scrub_stamp":"2024-09-13T10:01:49.922016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:57.799098+0000","scrub_duration":0.0015797980000000001,"objects_trimmed":0,"snaptrim_duration":0.00038734700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"459'2","reported_seq":490,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-13T10:01:59.945752+0000","last_change":"2024-09-13T10:01:59.945866+0000","last_active":"2024-09-13T10:01:59.945752+0000","last_peered":"2024-09-13T10:01:59.945752+0000","last_clean":"2024-09-13T10:01:59.945752+0000","last_became_active":"2024-09-13T09:23:50.440112+0000","last_became_peered":"2024-09-13T09:23:50.440112+0000","last_unstale":"2024-09-13T10:01:59.945752+0000","last_undegraded":"2024-09-13T10:01:59.945752+0000","last_fullsized":"2024-09-13T10:01:59.945752+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'2","last_scrub_stamp":"2024-09-13T10:01:59.945729+0000","last_deep_scrub":"459'2","last_deep_scrub_stamp":"2024-09-13T10:01:59.945729+0000","last_clean_scrub_stamp":"2024-09-13T10:01:59.945729+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:34.724567+0000","scrub_duration":0.002812443,"objects_trimmed":0,"snaptrim_duration":0.00048773600000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"459'99","reported_seq":616,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-13T10:01:48.919365+0000","last_change":"2024-09-13T10:01:48.919365+0000","last_active":"2024-09-13T10:01:48.919365+0000","last_peered":"2024-09-13T10:01:48.919365+0000","last_clean":"2024-09-13T10:01:48.919365+0000","last_became_active":"2024-09-13T09:23:45.886500+0000","last_became_peered":"2024-09-13T09:23:45.886500+0000","last_unstale":"2024-09-13T10:01:48.919365+0000","last_undegraded":"2024-09-13T10:01:48.919365+0000","last_fullsized":"2024-09-13T10:01:48.919365+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'99","last_scrub_stamp":"2024-09-13T10:01:48.919247+0000","last_deep_scrub":"459'99","last_deep_scrub_stamp":"2024-09-13T10:01:48.919247+0000","last_clean_scrub_stamp":"2024-09-13T10:01:48.919247+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:50:18.539681+0000","scrub_duration":0.0036986100000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738839,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1300476,"kb_used_data":5300,"kb_used_omap":1,"kb_used_meta":1295166,"kb_avail":93071364,"statfs":{"total":96636764160,"available":95305076736,"internally_reserved":0,"allocated":5427200,"data_stored":2656247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1326250438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:01:53 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.56200000000000006,"15min":0.46800000000000003},"min":{"1min":0.19500000000000001,"5min":0.159,"15min":0.14099999999999999},"max":{"1min":0.69199999999999995,"5min":7.9580000000000002,"15min":10.542},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.63400000000000001,"15min":0.53400000000000003},"min":{"1min":0.26100000000000001,"5min":0.20699999999999999,"15min":0.14299999999999999},"max":{"1min":0.71599999999999997,"5min":7.9909999999999997,"15min":10.577999999999999},"last":0.32500000000000001}]},{"osd":1,"last update":"Fri Sep 13 10:01:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47799999999999998,"15min":0.437},"min":{"1min":0.26900000000000002,"5min":0.183,"15min":0.16500000000000001},"max":{"1min":2.6160000000000001,"5min":4.4210000000000003,"15min":4.6429999999999998},"last":0.504},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.70999999999999996,"15min":0.57999999999999996},"min":{"1min":0.215,"5min":0.215,"15min":0.16900000000000001},"max":{"1min":2.4860000000000002,"5min":9.5009999999999994,"15min":9.8219999999999992},"last":0.40799999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738836,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1441532,"kb_used_data":5364,"kb_used_omap":1,"kb_used_meta":1436158,"kb_avail":92930308,"statfs":{"total":96636764160,"available":95160635392,"internally_reserved":0,"allocated":5492736,"data_stored":3246634,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1470626246},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:01:43 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.71299999999999997,"15min":0.51700000000000002},"min":{"1min":0.214,"5min":0.14299999999999999,"15min":0.13500000000000001},"max":{"1min":1.0449999999999999,"5min":11.082000000000001,"15min":11.082000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.371,"5min":0.53600000000000003,"15min":0.46800000000000003},"min":{"1min":0.25,"5min":0.20399999999999999,"15min":0.17599999999999999},"max":{"1min":1.0229999999999999,"5min":5.6059999999999999,"15min":6.9790000000000001},"last":0.32100000000000001}]},{"osd":2,"last update":"Fri Sep 13 10:01:43 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.66800000000000004,"15min":0.48999999999999999},"min":{"1min":0.23400000000000001,"5min":0.186,"15min":0.16900000000000001},"max":{"1min":1.1259999999999999,"5min":11.034000000000001,"15min":11.034000000000001},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.30099999999999999,"5min":0.45500000000000002,"15min":0.40699999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.44,"5min":3.835,"15min":3.835},"last":0.26000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738837,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1620284,"kb_used_data":5364,"kb_used_omap":1,"kb_used_meta":1614910,"kb_avail":92751556,"statfs":{"total":96636764160,"available":94977593344,"internally_reserved":0,"allocated":5492736,"data_stored":3246634,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1653668294},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 10:02:12 2024","interfaces":[{"interface":"back","average":{"1min":0.28399999999999997,"5min":0.69899999999999995,"15min":0.47199999999999998},"min":{"1min":0.19700000000000001,"5min":0.16400000000000001,"15min":0.14399999999999999},"max":{"1min":0.41299999999999998,"5min":10.384,"15min":10.384},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.75900000000000001,"15min":0.52500000000000002},"min":{"1min":0.25600000000000001,"5min":0.20300000000000001,"15min":0.17799999999999999},"max":{"1min":0.48499999999999999,"5min":10.427,"15min":10.427},"last":0.33200000000000002}]},{"osd":2,"last update":"Fri Sep 13 10:02:12 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.83999999999999997,"15min":0.50900000000000001},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.16200000000000001},"max":{"1min":0.47299999999999998,"5min":14.365,"15min":14.365},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.90200000000000002,"15min":0.56200000000000006},"min":{"1min":0.254,"5min":0.24099999999999999,"15min":0.19700000000000001},"max":{"1min":0.52600000000000002,"5min":14.401,"15min":14.401},"last":0.34899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T10:02:29.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-09-13T10:02:29.305 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-13T10:02:29.305 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-13T10:02:29.306 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-09-13T10:02:29.306 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T10:02:29.376 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-09-13T10:02:29.376 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-09-13T10:02:29.377 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T10:02:29.440 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-09-13T10:02:29.441 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-09-13T10:02:29.441 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T10:02:29.518 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-09-13T10:02:29.518 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-13T10:02:29.519 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-09-13T10:02:29.519 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T10:02:29.548 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-09-13T10:02:29.548 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-13T10:02:29.548 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-09-13T10:02:29.548 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T10:02:29.561 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-09-13T10:02:29.561 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-13T10:02:29.562 DEBUG:teuthology.orchestra.run.smithi190:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-13T10:02:29.664 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-13T10:02:29.664 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-09-13T10:02:29.945 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-13T10:02:29.945 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-09-13T10:02:30.080 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-13T10:02:30.081 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-09-13T10:02:30.207 INFO:tasks.ceph:Archiving mon data... 2024-09-13T10:02:30.208 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7901034/data/cluster2.mon.a.tgz 2024-09-13T10:02:30.209 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-09-13T10:02:30.224 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.zs06xu7yzG 2024-09-13T10:02:30.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.zs06xu7yzG 2024-09-13T10:02:30.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.zs06xu7yzG 2024-09-13T10:02:30.504 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.zs06xu7yzG is 444KB 2024-09-13T10:02:30.581 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.zs06xu7yzG 2024-09-13T10:02:30.597 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-13T10:02:30.597 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T10:02:30.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T10:02:30.665 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-13T10:02:30.679 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-13T10:02:30.679 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-13T10:02:30.953 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T10:02:30.954 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-09-13T10:02:30.966 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":1641,"stamp":"2024-09-13T10:02:29.142344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":283404,"kb_used_data":11236,"kb_used_omap":4,"kb_used_meta":272123,"kb_avail":282832116,"statfs":{"total":289910292480,"available":289620086784,"internally_reserved":0,"allocated":11505664,"data_stored":7276533,"data_compressed":59912,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":278654290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002222"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:01:42.826317+0000","last_change":"2024-09-13T09:23:14.650263+0000","last_active":"2024-09-13T10:01:42.826317+0000","last_peered":"2024-09-13T10:01:42.826317+0000","last_clean":"2024-09-13T10:01:42.826317+0000","last_became_active":"2024-09-13T09:23:12.644939+0000","last_became_peered":"2024-09-13T09:23:12.644939+0000","last_unstale":"2024-09-13T10:01:42.826317+0000","last_undegraded":"2024-09-13T10:01:42.826317+0000","last_fullsized":"2024-09-13T10:01:42.826317+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:32:52.401407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075243300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:01:42.830433+0000","last_change":"2024-09-13T09:23:14.650022+0000","last_active":"2024-09-13T10:01:42.830433+0000","last_peered":"2024-09-13T10:01:42.830433+0000","last_clean":"2024-09-13T10:01:42.830433+0000","last_became_active":"2024-09-13T09:23:12.646762+0000","last_became_peered":"2024-09-13T09:23:12.646762+0000","last_unstale":"2024-09-13T10:01:42.830433+0000","last_undegraded":"2024-09-13T10:01:42.830433+0000","last_fullsized":"2024-09-13T10:01:42.830433+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:01:59.548841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051643700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":478,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:40.811594+0000","last_change":"2024-09-13T09:23:14.649764+0000","last_active":"2024-09-13T10:01:40.811594+0000","last_peered":"2024-09-13T10:01:40.811594+0000","last_clean":"2024-09-13T10:01:40.811594+0000","last_became_active":"2024-09-13T09:23:12.644747+0000","last_became_peered":"2024-09-13T09:23:12.644747+0000","last_unstale":"2024-09-13T10:01:40.811594+0000","last_undegraded":"2024-09-13T10:01:40.811594+0000","last_fullsized":"2024-09-13T10:01:40.811594+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:59.606001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044772799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":478,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:40.811306+0000","last_change":"2024-09-13T09:23:14.650026+0000","last_active":"2024-09-13T10:01:40.811306+0000","last_peered":"2024-09-13T10:01:40.811306+0000","last_clean":"2024-09-13T10:01:40.811306+0000","last_became_active":"2024-09-13T09:23:12.644527+0000","last_became_peered":"2024-09-13T09:23:12.644527+0000","last_unstale":"2024-09-13T10:01:40.811306+0000","last_undegraded":"2024-09-13T10:01:40.811306+0000","last_fullsized":"2024-09-13T10:01:40.811306+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:37:34.744466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071815500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":481,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:40.771337+0000","last_change":"2024-09-13T09:23:14.643542+0000","last_active":"2024-09-13T10:01:40.771337+0000","last_peered":"2024-09-13T10:01:40.771337+0000","last_clean":"2024-09-13T10:01:40.771337+0000","last_became_active":"2024-09-13T09:23:12.646192+0000","last_became_peered":"2024-09-13T09:23:12.646192+0000","last_unstale":"2024-09-13T10:01:40.771337+0000","last_undegraded":"2024-09-13T10:01:40.771337+0000","last_fullsized":"2024-09-13T10:01:40.771337+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:48:00.021353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035806400000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":478,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:41.127323+0000","last_change":"2024-09-13T09:23:14.982720+0000","last_active":"2024-09-13T10:01:41.127323+0000","last_peered":"2024-09-13T10:01:41.127323+0000","last_clean":"2024-09-13T10:01:41.127323+0000","last_became_active":"2024-09-13T09:23:12.645432+0000","last_became_peered":"2024-09-13T09:23:12.645432+0000","last_unstale":"2024-09-13T10:01:41.127323+0000","last_undegraded":"2024-09-13T10:01:41.127323+0000","last_fullsized":"2024-09-13T10:01:41.127323+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:40:58.609609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036873199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":479,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:41.129187+0000","last_change":"2024-09-13T09:23:14.982718+0000","last_active":"2024-09-13T10:01:41.129187+0000","last_peered":"2024-09-13T10:01:41.129187+0000","last_clean":"2024-09-13T10:01:41.129187+0000","last_became_active":"2024-09-13T09:23:12.645717+0000","last_became_peered":"2024-09-13T09:23:12.645717+0000","last_unstale":"2024-09-13T10:01:41.129187+0000","last_undegraded":"2024-09-13T10:01:41.129187+0000","last_fullsized":"2024-09-13T10:01:41.129187+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:55.805671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037606,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":479,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:40.816431+0000","last_change":"2024-09-13T09:23:14.649739+0000","last_active":"2024-09-13T10:01:40.816431+0000","last_peered":"2024-09-13T10:01:40.816431+0000","last_clean":"2024-09-13T10:01:40.816431+0000","last_became_active":"2024-09-13T09:23:12.643941+0000","last_became_peered":"2024-09-13T09:23:12.643941+0000","last_unstale":"2024-09-13T10:01:40.816431+0000","last_undegraded":"2024-09-13T10:01:40.816431+0000","last_fullsized":"2024-09-13T10:01:40.816431+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:09.200514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041677400000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":604,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:40.814690+0000","last_change":"2024-09-13T09:23:07.618288+0000","last_active":"2024-09-13T10:01:40.814690+0000","last_peered":"2024-09-13T10:01:40.814690+0000","last_clean":"2024-09-13T10:01:40.814690+0000","last_became_active":"2024-09-13T09:23:07.617395+0000","last_became_peered":"2024-09-13T09:23:07.617395+0000","last_unstale":"2024-09-13T10:01:40.814690+0000","last_undegraded":"2024-09-13T10:01:40.814690+0000","last_fullsized":"2024-09-13T10:01:40.814690+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:06.608062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:06.608062+0000","last_clean_scrub_stamp":"2024-09-13T09:23:06.608062+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:39.950161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59912,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738845,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93000,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":89278,"kb_avail":94278840,"statfs":{"total":96636764160,"available":96541532160,"internally_reserved":0,"allocated":3792896,"data_stored":2032126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91421126},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.311,"15min":0.32000000000000001},"min":{"1min":0.219,"5min":0.19500000000000001,"15min":0.14799999999999999},"max":{"1min":0.45400000000000001,"5min":0.45900000000000002,"15min":1.1539999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.34699999999999998},"min":{"1min":0.30199999999999999,"5min":0.189,"15min":0.17599999999999999},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":1.0940000000000001},"last":0.378}]},{"osd":1,"last update":"Fri Sep 13 10:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.309,"15min":0.312},"min":{"1min":0.23300000000000001,"5min":0.218,"15min":0.17899999999999999},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.80700000000000005},"last":0.313},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.27200000000000002,"5min":0.223,"15min":0.17799999999999999},"max":{"1min":0.48599999999999999,"5min":0.50800000000000001,"15min":1.091},"last":0.34899999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738844,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66884,"kb_used_data":3768,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304956,"statfs":{"total":96636764160,"available":96568274944,"internally_reserved":0,"allocated":3858432,"data_stored":2622513,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:02:23 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.314,"15min":0.31},"min":{"1min":0.245,"5min":0.20399999999999999,"15min":0.16},"max":{"1min":0.44400000000000001,"5min":0.44800000000000001,"15min":1.163},"last":0.315},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.31,"15min":0.32100000000000001},"min":{"1min":0.26100000000000001,"5min":0.21299999999999999,"15min":0.17999999999999999},"max":{"1min":0.438,"5min":0.45900000000000002,"15min":3.7879999999999998},"last":0.26100000000000001}]},{"osd":2,"last update":"Fri Sep 13 10:02:23 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.32700000000000001,"15min":0.32200000000000001},"min":{"1min":0.27700000000000002,"5min":0.22500000000000001,"15min":0.17000000000000001},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.49199999999999999},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.32300000000000001,"15min":0.32600000000000001},"min":{"1min":0.26800000000000002,"5min":0.22,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.63600000000000001},"last":0.29099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738845,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":123520,"kb_used_data":3764,"kb_used_omap":1,"kb_used_meta":119742,"kb_avail":94248320,"statfs":{"total":96636764160,"available":96510279680,"internally_reserved":0,"allocated":3854336,"data_stored":2621894,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":122616262},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 10:01:25 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.30499999999999999,"15min":0.30599999999999999},"min":{"1min":0.22,"5min":0.20200000000000001,"15min":0.16900000000000001},"max":{"1min":0.41399999999999998,"5min":0.505,"15min":2.911},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.33300000000000002,"15min":0.32400000000000001},"min":{"1min":0.248,"5min":0.23799999999999999,"15min":0.19500000000000001},"max":{"1min":0.47899999999999998,"5min":0.49299999999999999,"15min":1.044},"last":0.372}]},{"osd":2,"last update":"Fri Sep 13 10:01:25 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.316,"15min":0.31},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.182},"max":{"1min":0.44500000000000001,"5min":0.47199999999999998,"15min":2.0459999999999998},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.32400000000000001,"15min":0.33900000000000002},"min":{"1min":0.23300000000000001,"5min":0.224,"15min":0.20899999999999999},"max":{"1min":0.443,"5min":0.48099999999999998,"15min":2.4089999999999998},"last":0.39000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T10:02:30.967 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-13T10:02:31.233 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T10:02:31.234 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-09-13T10:02:31.248 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":1642,"stamp":"2024-09-13T10:02:31.142642+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":283404,"kb_used_data":11236,"kb_used_omap":4,"kb_used_meta":272123,"kb_avail":282832116,"statfs":{"total":289910292480,"available":289620086784,"internally_reserved":0,"allocated":11505664,"data_stored":7276533,"data_compressed":59912,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":278654290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002199"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:01:42.826317+0000","last_change":"2024-09-13T09:23:14.650263+0000","last_active":"2024-09-13T10:01:42.826317+0000","last_peered":"2024-09-13T10:01:42.826317+0000","last_clean":"2024-09-13T10:01:42.826317+0000","last_became_active":"2024-09-13T09:23:12.644939+0000","last_became_peered":"2024-09-13T09:23:12.644939+0000","last_unstale":"2024-09-13T10:01:42.826317+0000","last_undegraded":"2024-09-13T10:01:42.826317+0000","last_fullsized":"2024-09-13T10:01:42.826317+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:32:52.401407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075243300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:01:42.830433+0000","last_change":"2024-09-13T09:23:14.650022+0000","last_active":"2024-09-13T10:01:42.830433+0000","last_peered":"2024-09-13T10:01:42.830433+0000","last_clean":"2024-09-13T10:01:42.830433+0000","last_became_active":"2024-09-13T09:23:12.646762+0000","last_became_peered":"2024-09-13T09:23:12.646762+0000","last_unstale":"2024-09-13T10:01:42.830433+0000","last_undegraded":"2024-09-13T10:01:42.830433+0000","last_fullsized":"2024-09-13T10:01:42.830433+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:01:59.548841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051643700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":478,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:40.811594+0000","last_change":"2024-09-13T09:23:14.649764+0000","last_active":"2024-09-13T10:01:40.811594+0000","last_peered":"2024-09-13T10:01:40.811594+0000","last_clean":"2024-09-13T10:01:40.811594+0000","last_became_active":"2024-09-13T09:23:12.644747+0000","last_became_peered":"2024-09-13T09:23:12.644747+0000","last_unstale":"2024-09-13T10:01:40.811594+0000","last_undegraded":"2024-09-13T10:01:40.811594+0000","last_fullsized":"2024-09-13T10:01:40.811594+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:59.606001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044772799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":478,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:40.811306+0000","last_change":"2024-09-13T09:23:14.650026+0000","last_active":"2024-09-13T10:01:40.811306+0000","last_peered":"2024-09-13T10:01:40.811306+0000","last_clean":"2024-09-13T10:01:40.811306+0000","last_became_active":"2024-09-13T09:23:12.644527+0000","last_became_peered":"2024-09-13T09:23:12.644527+0000","last_unstale":"2024-09-13T10:01:40.811306+0000","last_undegraded":"2024-09-13T10:01:40.811306+0000","last_fullsized":"2024-09-13T10:01:40.811306+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:37:34.744466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071815500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":481,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:40.771337+0000","last_change":"2024-09-13T09:23:14.643542+0000","last_active":"2024-09-13T10:01:40.771337+0000","last_peered":"2024-09-13T10:01:40.771337+0000","last_clean":"2024-09-13T10:01:40.771337+0000","last_became_active":"2024-09-13T09:23:12.646192+0000","last_became_peered":"2024-09-13T09:23:12.646192+0000","last_unstale":"2024-09-13T10:01:40.771337+0000","last_undegraded":"2024-09-13T10:01:40.771337+0000","last_fullsized":"2024-09-13T10:01:40.771337+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:48:00.021353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035806400000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":478,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:41.127323+0000","last_change":"2024-09-13T09:23:14.982720+0000","last_active":"2024-09-13T10:01:41.127323+0000","last_peered":"2024-09-13T10:01:41.127323+0000","last_clean":"2024-09-13T10:01:41.127323+0000","last_became_active":"2024-09-13T09:23:12.645432+0000","last_became_peered":"2024-09-13T09:23:12.645432+0000","last_unstale":"2024-09-13T10:01:41.127323+0000","last_undegraded":"2024-09-13T10:01:41.127323+0000","last_fullsized":"2024-09-13T10:01:41.127323+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:40:58.609609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036873199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":479,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:41.129187+0000","last_change":"2024-09-13T09:23:14.982718+0000","last_active":"2024-09-13T10:01:41.129187+0000","last_peered":"2024-09-13T10:01:41.129187+0000","last_clean":"2024-09-13T10:01:41.129187+0000","last_became_active":"2024-09-13T09:23:12.645717+0000","last_became_peered":"2024-09-13T09:23:12.645717+0000","last_unstale":"2024-09-13T10:01:41.129187+0000","last_undegraded":"2024-09-13T10:01:41.129187+0000","last_fullsized":"2024-09-13T10:01:41.129187+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:55.805671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037606,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":479,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:40.816431+0000","last_change":"2024-09-13T09:23:14.649739+0000","last_active":"2024-09-13T10:01:40.816431+0000","last_peered":"2024-09-13T10:01:40.816431+0000","last_clean":"2024-09-13T10:01:40.816431+0000","last_became_active":"2024-09-13T09:23:12.643941+0000","last_became_peered":"2024-09-13T09:23:12.643941+0000","last_unstale":"2024-09-13T10:01:40.816431+0000","last_undegraded":"2024-09-13T10:01:40.816431+0000","last_fullsized":"2024-09-13T10:01:40.816431+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:09.200514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041677400000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":604,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:40.814690+0000","last_change":"2024-09-13T09:23:07.618288+0000","last_active":"2024-09-13T10:01:40.814690+0000","last_peered":"2024-09-13T10:01:40.814690+0000","last_clean":"2024-09-13T10:01:40.814690+0000","last_became_active":"2024-09-13T09:23:07.617395+0000","last_became_peered":"2024-09-13T09:23:07.617395+0000","last_unstale":"2024-09-13T10:01:40.814690+0000","last_undegraded":"2024-09-13T10:01:40.814690+0000","last_fullsized":"2024-09-13T10:01:40.814690+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:06.608062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:06.608062+0000","last_clean_scrub_stamp":"2024-09-13T09:23:06.608062+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:39.950161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59912,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738846,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93000,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":89278,"kb_avail":94278840,"statfs":{"total":96636764160,"available":96541532160,"internally_reserved":0,"allocated":3792896,"data_stored":2032126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91421126},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.311,"15min":0.32000000000000001},"min":{"1min":0.219,"5min":0.19500000000000001,"15min":0.14799999999999999},"max":{"1min":0.45400000000000001,"5min":0.45900000000000002,"15min":1.1539999999999999},"last":0.312},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.34699999999999998},"min":{"1min":0.30199999999999999,"5min":0.189,"15min":0.17599999999999999},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":1.0940000000000001},"last":0.41899999999999998}]},{"osd":1,"last update":"Fri Sep 13 10:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.309,"15min":0.312},"min":{"1min":0.23300000000000001,"5min":0.218,"15min":0.17899999999999999},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.80700000000000005},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.27200000000000002,"5min":0.223,"15min":0.17799999999999999},"max":{"1min":0.48599999999999999,"5min":0.50800000000000001,"15min":1.091},"last":0.39100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738845,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66884,"kb_used_data":3768,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304956,"statfs":{"total":96636764160,"available":96568274944,"internally_reserved":0,"allocated":3858432,"data_stored":2622513,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:02:23 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.314,"15min":0.31},"min":{"1min":0.245,"5min":0.20399999999999999,"15min":0.16},"max":{"1min":0.44400000000000001,"5min":0.44800000000000001,"15min":1.163},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.31,"15min":0.32100000000000001},"min":{"1min":0.26100000000000001,"5min":0.21299999999999999,"15min":0.17999999999999999},"max":{"1min":0.438,"5min":0.45900000000000002,"15min":3.7879999999999998},"last":0.41899999999999998}]},{"osd":2,"last update":"Fri Sep 13 10:02:23 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.32700000000000001,"15min":0.32200000000000001},"min":{"1min":0.27700000000000002,"5min":0.22500000000000001,"15min":0.17000000000000001},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.49199999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.32300000000000001,"15min":0.32600000000000001},"min":{"1min":0.26800000000000002,"5min":0.22,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.63600000000000001},"last":0.436}]}]},{"osd":0,"up_from":8,"seq":34359738846,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":123520,"kb_used_data":3764,"kb_used_omap":1,"kb_used_meta":119742,"kb_avail":94248320,"statfs":{"total":96636764160,"available":96510279680,"internally_reserved":0,"allocated":3854336,"data_stored":2621894,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":122616262},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 10:02:25 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.309,"15min":0.308},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.16900000000000001},"max":{"1min":0.41599999999999998,"5min":0.441,"15min":2.911},"last":0.373},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.34000000000000002,"15min":0.32600000000000001},"min":{"1min":0.253,"5min":0.23799999999999999,"15min":0.19500000000000001},"max":{"1min":0.46300000000000002,"5min":0.49299999999999999,"15min":1.044},"last":0.40899999999999997}]},{"osd":2,"last update":"Fri Sep 13 10:02:25 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.312},"min":{"1min":0.24299999999999999,"5min":0.20799999999999999,"15min":0.182},"max":{"1min":0.43099999999999999,"5min":0.47199999999999998,"15min":2.0459999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.32800000000000001,"15min":0.34000000000000002},"min":{"1min":0.26100000000000001,"5min":0.224,"15min":0.20899999999999999},"max":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":2.4089999999999998},"last":0.38600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T10:02:31.249 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-13T10:02:31.249 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-13T10:02:31.514 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T10:02:31.515 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-09-13T10:02:31.526 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":1642,"stamp":"2024-09-13T10:02:31.142642+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":283404,"kb_used_data":11236,"kb_used_omap":4,"kb_used_meta":272123,"kb_avail":282832116,"statfs":{"total":289910292480,"available":289620086784,"internally_reserved":0,"allocated":11505664,"data_stored":7276533,"data_compressed":59912,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":278654290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002199"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:01:42.826317+0000","last_change":"2024-09-13T09:23:14.650263+0000","last_active":"2024-09-13T10:01:42.826317+0000","last_peered":"2024-09-13T10:01:42.826317+0000","last_clean":"2024-09-13T10:01:42.826317+0000","last_became_active":"2024-09-13T09:23:12.644939+0000","last_became_peered":"2024-09-13T09:23:12.644939+0000","last_unstale":"2024-09-13T10:01:42.826317+0000","last_undegraded":"2024-09-13T10:01:42.826317+0000","last_fullsized":"2024-09-13T10:01:42.826317+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:32:52.401407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075243300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:01:42.830433+0000","last_change":"2024-09-13T09:23:14.650022+0000","last_active":"2024-09-13T10:01:42.830433+0000","last_peered":"2024-09-13T10:01:42.830433+0000","last_clean":"2024-09-13T10:01:42.830433+0000","last_became_active":"2024-09-13T09:23:12.646762+0000","last_became_peered":"2024-09-13T09:23:12.646762+0000","last_unstale":"2024-09-13T10:01:42.830433+0000","last_undegraded":"2024-09-13T10:01:42.830433+0000","last_fullsized":"2024-09-13T10:01:42.830433+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:01:59.548841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051643700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":478,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:40.811594+0000","last_change":"2024-09-13T09:23:14.649764+0000","last_active":"2024-09-13T10:01:40.811594+0000","last_peered":"2024-09-13T10:01:40.811594+0000","last_clean":"2024-09-13T10:01:40.811594+0000","last_became_active":"2024-09-13T09:23:12.644747+0000","last_became_peered":"2024-09-13T09:23:12.644747+0000","last_unstale":"2024-09-13T10:01:40.811594+0000","last_undegraded":"2024-09-13T10:01:40.811594+0000","last_fullsized":"2024-09-13T10:01:40.811594+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:59.606001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044772799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":478,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:40.811306+0000","last_change":"2024-09-13T09:23:14.650026+0000","last_active":"2024-09-13T10:01:40.811306+0000","last_peered":"2024-09-13T10:01:40.811306+0000","last_clean":"2024-09-13T10:01:40.811306+0000","last_became_active":"2024-09-13T09:23:12.644527+0000","last_became_peered":"2024-09-13T09:23:12.644527+0000","last_unstale":"2024-09-13T10:01:40.811306+0000","last_undegraded":"2024-09-13T10:01:40.811306+0000","last_fullsized":"2024-09-13T10:01:40.811306+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:37:34.744466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071815500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":481,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:40.771337+0000","last_change":"2024-09-13T09:23:14.643542+0000","last_active":"2024-09-13T10:01:40.771337+0000","last_peered":"2024-09-13T10:01:40.771337+0000","last_clean":"2024-09-13T10:01:40.771337+0000","last_became_active":"2024-09-13T09:23:12.646192+0000","last_became_peered":"2024-09-13T09:23:12.646192+0000","last_unstale":"2024-09-13T10:01:40.771337+0000","last_undegraded":"2024-09-13T10:01:40.771337+0000","last_fullsized":"2024-09-13T10:01:40.771337+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:48:00.021353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035806400000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":478,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:41.127323+0000","last_change":"2024-09-13T09:23:14.982720+0000","last_active":"2024-09-13T10:01:41.127323+0000","last_peered":"2024-09-13T10:01:41.127323+0000","last_clean":"2024-09-13T10:01:41.127323+0000","last_became_active":"2024-09-13T09:23:12.645432+0000","last_became_peered":"2024-09-13T09:23:12.645432+0000","last_unstale":"2024-09-13T10:01:41.127323+0000","last_undegraded":"2024-09-13T10:01:41.127323+0000","last_fullsized":"2024-09-13T10:01:41.127323+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:40:58.609609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036873199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":479,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:41.129187+0000","last_change":"2024-09-13T09:23:14.982718+0000","last_active":"2024-09-13T10:01:41.129187+0000","last_peered":"2024-09-13T10:01:41.129187+0000","last_clean":"2024-09-13T10:01:41.129187+0000","last_became_active":"2024-09-13T09:23:12.645717+0000","last_became_peered":"2024-09-13T09:23:12.645717+0000","last_unstale":"2024-09-13T10:01:41.129187+0000","last_undegraded":"2024-09-13T10:01:41.129187+0000","last_fullsized":"2024-09-13T10:01:41.129187+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:55.805671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037606,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":479,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:40.816431+0000","last_change":"2024-09-13T09:23:14.649739+0000","last_active":"2024-09-13T10:01:40.816431+0000","last_peered":"2024-09-13T10:01:40.816431+0000","last_clean":"2024-09-13T10:01:40.816431+0000","last_became_active":"2024-09-13T09:23:12.643941+0000","last_became_peered":"2024-09-13T09:23:12.643941+0000","last_unstale":"2024-09-13T10:01:40.816431+0000","last_undegraded":"2024-09-13T10:01:40.816431+0000","last_fullsized":"2024-09-13T10:01:40.816431+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:09.200514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041677400000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":604,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:40.814690+0000","last_change":"2024-09-13T09:23:07.618288+0000","last_active":"2024-09-13T10:01:40.814690+0000","last_peered":"2024-09-13T10:01:40.814690+0000","last_clean":"2024-09-13T10:01:40.814690+0000","last_became_active":"2024-09-13T09:23:07.617395+0000","last_became_peered":"2024-09-13T09:23:07.617395+0000","last_unstale":"2024-09-13T10:01:40.814690+0000","last_undegraded":"2024-09-13T10:01:40.814690+0000","last_fullsized":"2024-09-13T10:01:40.814690+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:06.608062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:06.608062+0000","last_clean_scrub_stamp":"2024-09-13T09:23:06.608062+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:39.950161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59912,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738846,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93000,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":89278,"kb_avail":94278840,"statfs":{"total":96636764160,"available":96541532160,"internally_reserved":0,"allocated":3792896,"data_stored":2032126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91421126},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.311,"15min":0.32000000000000001},"min":{"1min":0.219,"5min":0.19500000000000001,"15min":0.14799999999999999},"max":{"1min":0.45400000000000001,"5min":0.45900000000000002,"15min":1.1539999999999999},"last":0.312},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.34699999999999998},"min":{"1min":0.30199999999999999,"5min":0.189,"15min":0.17599999999999999},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":1.0940000000000001},"last":0.41899999999999998}]},{"osd":1,"last update":"Fri Sep 13 10:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.309,"15min":0.312},"min":{"1min":0.23300000000000001,"5min":0.218,"15min":0.17899999999999999},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.80700000000000005},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.27200000000000002,"5min":0.223,"15min":0.17799999999999999},"max":{"1min":0.48599999999999999,"5min":0.50800000000000001,"15min":1.091},"last":0.39100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738845,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66884,"kb_used_data":3768,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304956,"statfs":{"total":96636764160,"available":96568274944,"internally_reserved":0,"allocated":3858432,"data_stored":2622513,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:02:23 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.314,"15min":0.31},"min":{"1min":0.245,"5min":0.20399999999999999,"15min":0.16},"max":{"1min":0.44400000000000001,"5min":0.44800000000000001,"15min":1.163},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.31,"15min":0.32100000000000001},"min":{"1min":0.26100000000000001,"5min":0.21299999999999999,"15min":0.17999999999999999},"max":{"1min":0.438,"5min":0.45900000000000002,"15min":3.7879999999999998},"last":0.41899999999999998}]},{"osd":2,"last update":"Fri Sep 13 10:02:23 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.32700000000000001,"15min":0.32200000000000001},"min":{"1min":0.27700000000000002,"5min":0.22500000000000001,"15min":0.17000000000000001},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.49199999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.32300000000000001,"15min":0.32600000000000001},"min":{"1min":0.26800000000000002,"5min":0.22,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.63600000000000001},"last":0.436}]}]},{"osd":0,"up_from":8,"seq":34359738846,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":123520,"kb_used_data":3764,"kb_used_omap":1,"kb_used_meta":119742,"kb_avail":94248320,"statfs":{"total":96636764160,"available":96510279680,"internally_reserved":0,"allocated":3854336,"data_stored":2621894,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":122616262},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 10:02:25 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.309,"15min":0.308},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.16900000000000001},"max":{"1min":0.41599999999999998,"5min":0.441,"15min":2.911},"last":0.373},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.34000000000000002,"15min":0.32600000000000001},"min":{"1min":0.253,"5min":0.23799999999999999,"15min":0.19500000000000001},"max":{"1min":0.46300000000000002,"5min":0.49299999999999999,"15min":1.044},"last":0.40899999999999997}]},{"osd":2,"last update":"Fri Sep 13 10:02:25 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.312},"min":{"1min":0.24299999999999999,"5min":0.20799999999999999,"15min":0.182},"max":{"1min":0.43099999999999999,"5min":0.47199999999999998,"15min":2.0459999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.32800000000000001,"15min":0.34000000000000002},"min":{"1min":0.26100000000000001,"5min":0.224,"15min":0.20899999999999999},"max":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":2.4089999999999998},"last":0.38600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T10:02:31.527 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-13T10:02:31.797 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T10:02:31.797 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":461,"fsid":"b196829d-0abf-49d7-afed-e85b83e5f85c","created":"2024-09-13T09:22:56.382934+0000","modified":"2024-09-13T10:01:42.763436+0000","last_up_change":"2024-09-13T09:23:04.592007+0000","last_in_change":"2024-09-13T09:22:57.253244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T09:23:06.568388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T09:23:11.422917+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6a665a2f-8cfa-47ec-802a-18764b684093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6805","nonce":3789617865},{"type":"v1","addr":"172.21.15.122:6807","nonce":3789617865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6809","nonce":3789617865},{"type":"v1","addr":"172.21.15.122:6811","nonce":3789617865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3789617865},{"type":"v1","addr":"172.21.15.122:6815","nonce":3789617865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3789617865},{"type":"v1","addr":"172.21.15.122:6813","nonce":3789617865}]},"public_addr":"172.21.15.122:6807/3789617865","cluster_addr":"172.21.15.122:6811/3789617865","heartbeat_back_addr":"172.21.15.122:6815/3789617865","heartbeat_front_addr":"172.21.15.122:6813/3789617865","state":["exists","up"]},{"osd":1,"uuid":"4f51e85c-4d33-4de7-9136-233fc8b9e47e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1208258757},{"type":"v1","addr":"172.21.15.122:6817","nonce":1208258757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1208258757},{"type":"v1","addr":"172.21.15.122:6819","nonce":1208258757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1208258757},{"type":"v1","addr":"172.21.15.122:6823","nonce":1208258757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1208258757},{"type":"v1","addr":"172.21.15.122:6821","nonce":1208258757}]},"public_addr":"172.21.15.122:6817/1208258757","cluster_addr":"172.21.15.122:6819/1208258757","heartbeat_back_addr":"172.21.15.122:6823/1208258757","heartbeat_front_addr":"172.21.15.122:6821/1208258757","state":["exists","up"]},{"osd":2,"uuid":"73635564-c0af-437e-8216-f4bc6a1f4c59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1711088402},{"type":"v1","addr":"172.21.15.122:6801","nonce":1711088402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1711088402},{"type":"v1","addr":"172.21.15.122:6803","nonce":1711088402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1711088402},{"type":"v1","addr":"172.21.15.122:6810","nonce":1711088402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1711088402},{"type":"v1","addr":"172.21.15.122:6806","nonce":1711088402}]},"public_addr":"172.21.15.122:6801/1711088402","cluster_addr":"172.21.15.122:6803/1711088402","heartbeat_back_addr":"172.21.15.122:6810/1711088402","heartbeat_front_addr":"172.21.15.122:6806/1711088402","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:02.441197+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:02.504333+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T09:23:02.520433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T10:02:32.810 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-09-13T10:02:32.811 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-13T10:02:32.950 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T10:02:32.950 INFO:teuthology.orchestra.run.smithi122.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T10:02:32.950 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T10:02:32.961 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-09-13T10:02:33.232 INFO:teuthology.orchestra.run.smithi122.stderr:instructed osd(s) 0 to deep-scrub 2024-09-13T10:02:33.245 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-09-13T10:02:33.245 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-13T10:02:33.365 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T10:02:33.365 INFO:teuthology.orchestra.run.smithi122.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T10:02:33.365 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T10:02:33.375 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-09-13T10:02:33.645 INFO:teuthology.orchestra.run.smithi122.stderr:instructed osd(s) 1 to deep-scrub 2024-09-13T10:02:33.658 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-09-13T10:02:33.658 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-13T10:02:33.779 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T10:02:33.779 INFO:teuthology.orchestra.run.smithi122.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T10:02:33.779 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T10:02:33.790 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-09-13T10:02:34.061 INFO:teuthology.orchestra.run.smithi122.stderr:instructed osd(s) 2 to deep-scrub 2024-09-13T10:02:34.073 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-13T10:02:34.340 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T10:02:34.340 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-09-13T10:02:34.352 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":1643,"stamp":"2024-09-13T10:02:33.142931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":283404,"kb_used_data":11236,"kb_used_omap":4,"kb_used_meta":272123,"kb_avail":282832116,"statfs":{"total":289910292480,"available":289620086784,"internally_reserved":0,"allocated":11505664,"data_stored":7276533,"data_compressed":59912,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":278654290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002157"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:01:42.826317+0000","last_change":"2024-09-13T09:23:14.650263+0000","last_active":"2024-09-13T10:01:42.826317+0000","last_peered":"2024-09-13T10:01:42.826317+0000","last_clean":"2024-09-13T10:01:42.826317+0000","last_became_active":"2024-09-13T09:23:12.644939+0000","last_became_peered":"2024-09-13T09:23:12.644939+0000","last_unstale":"2024-09-13T10:01:42.826317+0000","last_undegraded":"2024-09-13T10:01:42.826317+0000","last_fullsized":"2024-09-13T10:01:42.826317+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:32:52.401407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075243300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":478,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:01:42.830433+0000","last_change":"2024-09-13T09:23:14.650022+0000","last_active":"2024-09-13T10:01:42.830433+0000","last_peered":"2024-09-13T10:01:42.830433+0000","last_clean":"2024-09-13T10:01:42.830433+0000","last_became_active":"2024-09-13T09:23:12.646762+0000","last_became_peered":"2024-09-13T09:23:12.646762+0000","last_unstale":"2024-09-13T10:01:42.830433+0000","last_undegraded":"2024-09-13T10:01:42.830433+0000","last_fullsized":"2024-09-13T10:01:42.830433+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:01:59.548841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051643700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":478,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:40.811594+0000","last_change":"2024-09-13T09:23:14.649764+0000","last_active":"2024-09-13T10:01:40.811594+0000","last_peered":"2024-09-13T10:01:40.811594+0000","last_clean":"2024-09-13T10:01:40.811594+0000","last_became_active":"2024-09-13T09:23:12.644747+0000","last_became_peered":"2024-09-13T09:23:12.644747+0000","last_unstale":"2024-09-13T10:01:40.811594+0000","last_undegraded":"2024-09-13T10:01:40.811594+0000","last_fullsized":"2024-09-13T10:01:40.811594+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:59.606001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044772799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":478,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:40.811306+0000","last_change":"2024-09-13T09:23:14.650026+0000","last_active":"2024-09-13T10:01:40.811306+0000","last_peered":"2024-09-13T10:01:40.811306+0000","last_clean":"2024-09-13T10:01:40.811306+0000","last_became_active":"2024-09-13T09:23:12.644527+0000","last_became_peered":"2024-09-13T09:23:12.644527+0000","last_unstale":"2024-09-13T10:01:40.811306+0000","last_undegraded":"2024-09-13T10:01:40.811306+0000","last_fullsized":"2024-09-13T10:01:40.811306+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:37:34.744466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071815500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":481,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:40.771337+0000","last_change":"2024-09-13T09:23:14.643542+0000","last_active":"2024-09-13T10:01:40.771337+0000","last_peered":"2024-09-13T10:01:40.771337+0000","last_clean":"2024-09-13T10:01:40.771337+0000","last_became_active":"2024-09-13T09:23:12.646192+0000","last_became_peered":"2024-09-13T09:23:12.646192+0000","last_unstale":"2024-09-13T10:01:40.771337+0000","last_undegraded":"2024-09-13T10:01:40.771337+0000","last_fullsized":"2024-09-13T10:01:40.771337+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:48:00.021353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035806400000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":478,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:41.127323+0000","last_change":"2024-09-13T09:23:14.982720+0000","last_active":"2024-09-13T10:01:41.127323+0000","last_peered":"2024-09-13T10:01:41.127323+0000","last_clean":"2024-09-13T10:01:41.127323+0000","last_became_active":"2024-09-13T09:23:12.645432+0000","last_became_peered":"2024-09-13T09:23:12.645432+0000","last_unstale":"2024-09-13T10:01:41.127323+0000","last_undegraded":"2024-09-13T10:01:41.127323+0000","last_fullsized":"2024-09-13T10:01:41.127323+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:40:58.609609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036873199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":479,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:41.129187+0000","last_change":"2024-09-13T09:23:14.982718+0000","last_active":"2024-09-13T10:01:41.129187+0000","last_peered":"2024-09-13T10:01:41.129187+0000","last_clean":"2024-09-13T10:01:41.129187+0000","last_became_active":"2024-09-13T09:23:12.645717+0000","last_became_peered":"2024-09-13T09:23:12.645717+0000","last_unstale":"2024-09-13T10:01:41.129187+0000","last_undegraded":"2024-09-13T10:01:41.129187+0000","last_fullsized":"2024-09-13T10:01:41.129187+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:55.805671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037606,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":479,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:40.816431+0000","last_change":"2024-09-13T09:23:14.649739+0000","last_active":"2024-09-13T10:01:40.816431+0000","last_peered":"2024-09-13T10:01:40.816431+0000","last_clean":"2024-09-13T10:01:40.816431+0000","last_became_active":"2024-09-13T09:23:12.643941+0000","last_became_peered":"2024-09-13T09:23:12.643941+0000","last_unstale":"2024-09-13T10:01:40.816431+0000","last_undegraded":"2024-09-13T10:01:40.816431+0000","last_fullsized":"2024-09-13T10:01:40.816431+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:09.200514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041677400000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":604,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-13T10:01:40.814690+0000","last_change":"2024-09-13T09:23:07.618288+0000","last_active":"2024-09-13T10:01:40.814690+0000","last_peered":"2024-09-13T10:01:40.814690+0000","last_clean":"2024-09-13T10:01:40.814690+0000","last_became_active":"2024-09-13T09:23:07.617395+0000","last_became_peered":"2024-09-13T09:23:07.617395+0000","last_unstale":"2024-09-13T10:01:40.814690+0000","last_undegraded":"2024-09-13T10:01:40.814690+0000","last_fullsized":"2024-09-13T10:01:40.814690+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:06.608062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:06.608062+0000","last_clean_scrub_stamp":"2024-09-13T09:23:06.608062+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:39.950161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59912,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738846,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93000,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":89278,"kb_avail":94278840,"statfs":{"total":96636764160,"available":96541532160,"internally_reserved":0,"allocated":3792896,"data_stored":2032126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91421126},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.311,"15min":0.32000000000000001},"min":{"1min":0.219,"5min":0.19500000000000001,"15min":0.14799999999999999},"max":{"1min":0.45400000000000001,"5min":0.45900000000000002,"15min":1.1539999999999999},"last":0.312},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.34699999999999998},"min":{"1min":0.30199999999999999,"5min":0.189,"15min":0.17599999999999999},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":1.0940000000000001},"last":0.41899999999999998}]},{"osd":1,"last update":"Fri Sep 13 10:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.309,"15min":0.312},"min":{"1min":0.23300000000000001,"5min":0.218,"15min":0.17899999999999999},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.80700000000000005},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.27200000000000002,"5min":0.223,"15min":0.17799999999999999},"max":{"1min":0.48599999999999999,"5min":0.50800000000000001,"15min":1.091},"last":0.39100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738845,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66884,"kb_used_data":3768,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304956,"statfs":{"total":96636764160,"available":96568274944,"internally_reserved":0,"allocated":3858432,"data_stored":2622513,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:02:23 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.314,"15min":0.31},"min":{"1min":0.245,"5min":0.20399999999999999,"15min":0.16},"max":{"1min":0.44400000000000001,"5min":0.44800000000000001,"15min":1.163},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.31,"15min":0.32100000000000001},"min":{"1min":0.26100000000000001,"5min":0.21299999999999999,"15min":0.17999999999999999},"max":{"1min":0.438,"5min":0.45900000000000002,"15min":3.7879999999999998},"last":0.41899999999999998}]},{"osd":2,"last update":"Fri Sep 13 10:02:23 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.32700000000000001,"15min":0.32200000000000001},"min":{"1min":0.27700000000000002,"5min":0.22500000000000001,"15min":0.17000000000000001},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.49199999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.32300000000000001,"15min":0.32600000000000001},"min":{"1min":0.26800000000000002,"5min":0.22,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.63600000000000001},"last":0.436}]}]},{"osd":0,"up_from":8,"seq":34359738846,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":123520,"kb_used_data":3764,"kb_used_omap":1,"kb_used_meta":119742,"kb_avail":94248320,"statfs":{"total":96636764160,"available":96510279680,"internally_reserved":0,"allocated":3854336,"data_stored":2621894,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":122616262},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 10:02:25 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.309,"15min":0.308},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.16900000000000001},"max":{"1min":0.41599999999999998,"5min":0.441,"15min":2.911},"last":0.373},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.34000000000000002,"15min":0.32600000000000001},"min":{"1min":0.253,"5min":0.23799999999999999,"15min":0.19500000000000001},"max":{"1min":0.46300000000000002,"5min":0.49299999999999999,"15min":1.044},"last":0.40899999999999997}]},{"osd":2,"last update":"Fri Sep 13 10:02:25 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.312},"min":{"1min":0.24299999999999999,"5min":0.20799999999999999,"15min":0.182},"max":{"1min":0.43099999999999999,"5min":0.47199999999999998,"15min":2.0459999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.32800000000000001,"15min":0.34000000000000002},"min":{"1min":0.26100000000000001,"5min":0.224,"15min":0.20899999999999999},"max":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":2.4089999999999998},"last":0.38600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T10:02:34.353 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-13T09:23:11.623377+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=9, tm_min=23, tm_sec=11, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=10, tm_min=2, tm_sec=31, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T10:02:34.353 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-13T09:23:11.623377+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=9, tm_min=23, tm_sec=11, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=10, tm_min=2, tm_sec=31, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T10:02:34.353 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-13T09:23:11.623377+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=9, tm_min=23, tm_sec=11, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=10, tm_min=2, tm_sec=31, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T10:02:34.353 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-13T09:23:11.623377+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=9, tm_min=23, tm_sec=11, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=10, tm_min=2, tm_sec=31, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T10:02:34.353 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-13T09:23:11.623377+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=9, tm_min=23, tm_sec=11, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=10, tm_min=2, tm_sec=31, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T10:02:34.354 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-13T09:23:11.623377+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=9, tm_min=23, tm_sec=11, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=10, tm_min=2, tm_sec=31, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T10:02:34.354 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-13T09:23:11.623377+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=9, tm_min=23, tm_sec=11, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=10, tm_min=2, tm_sec=31, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T10:02:34.354 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-13T09:23:11.623377+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=9, tm_min=23, tm_sec=11, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=10, tm_min=2, tm_sec=31, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T10:02:34.354 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-13T09:23:06.608062+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=9, tm_min=23, tm_sec=6, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=10, tm_min=2, tm_sec=31, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T10:02:34.354 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-13T10:02:54.356 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-13T10:02:54.635 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T10:02:54.635 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-09-13T10:02:54.647 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":1653,"stamp":"2024-09-13T10:02:53.146522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":262780,"kb_used_data":11236,"kb_used_omap":4,"kb_used_meta":251515,"kb_avail":282852740,"statfs":{"total":289910292480,"available":289641205760,"internally_reserved":0,"allocated":11505664,"data_stored":7276533,"data_compressed":59912,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":257551698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002170"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":479,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:02:14.707367+0000","last_change":"2024-09-13T09:23:14.650263+0000","last_active":"2024-09-13T10:02:14.707367+0000","last_peered":"2024-09-13T10:02:14.707367+0000","last_clean":"2024-09-13T10:02:14.707367+0000","last_became_active":"2024-09-13T09:23:12.644939+0000","last_became_peered":"2024-09-13T09:23:12.644939+0000","last_unstale":"2024-09-13T10:02:14.707367+0000","last_undegraded":"2024-09-13T10:02:14.707367+0000","last_fullsized":"2024-09-13T10:02:14.707367+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T09:23:11.623377+0000","last_clean_scrub_stamp":"2024-09-13T09:23:11.623377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075243300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":490,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:02:42.860198+0000","last_change":"2024-09-13T10:02:42.860319+0000","last_active":"2024-09-13T10:02:42.860198+0000","last_peered":"2024-09-13T10:02:42.860198+0000","last_clean":"2024-09-13T10:02:42.860198+0000","last_became_active":"2024-09-13T09:23:12.646762+0000","last_became_peered":"2024-09-13T09:23:12.646762+0000","last_unstale":"2024-09-13T10:02:42.860198+0000","last_undegraded":"2024-09-13T10:02:42.860198+0000","last_fullsized":"2024-09-13T10:02:42.860198+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:02:42.860176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:02:42.860176+0000","last_clean_scrub_stamp":"2024-09-13T10:02:42.860176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:28.485888+0000","scrub_duration":0.0013819220000000001,"objects_trimmed":0,"snaptrim_duration":0.00051643700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":490,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:02:37.830042+0000","last_change":"2024-09-13T10:02:37.830162+0000","last_active":"2024-09-13T10:02:37.830042+0000","last_peered":"2024-09-13T10:02:37.830042+0000","last_clean":"2024-09-13T10:02:37.830042+0000","last_became_active":"2024-09-13T09:23:12.644747+0000","last_became_peered":"2024-09-13T09:23:12.644747+0000","last_unstale":"2024-09-13T10:02:37.830042+0000","last_undegraded":"2024-09-13T10:02:37.830042+0000","last_fullsized":"2024-09-13T10:02:37.830042+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:02:37.830020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:02:37.830020+0000","last_clean_scrub_stamp":"2024-09-13T10:02:37.830020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:41.135864+0000","scrub_duration":0.001460469,"objects_trimmed":0,"snaptrim_duration":0.00044772799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":491,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:02:43.842275+0000","last_change":"2024-09-13T10:02:43.842275+0000","last_active":"2024-09-13T10:02:43.842275+0000","last_peered":"2024-09-13T10:02:43.842275+0000","last_clean":"2024-09-13T10:02:43.842275+0000","last_became_active":"2024-09-13T09:23:12.644527+0000","last_became_peered":"2024-09-13T09:23:12.644527+0000","last_unstale":"2024-09-13T10:02:43.842275+0000","last_undegraded":"2024-09-13T10:02:43.842275+0000","last_fullsized":"2024-09-13T10:02:43.842275+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:02:43.842073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:02:43.842073+0000","last_clean_scrub_stamp":"2024-09-13T10:02:43.842073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:03.453693+0000","scrub_duration":0.0019788430000000001,"objects_trimmed":0,"snaptrim_duration":0.00071815500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"461'3","reported_seq":495,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:02:34.861895+0000","last_change":"2024-09-13T10:02:34.861895+0000","last_active":"2024-09-13T10:02:34.861895+0000","last_peered":"2024-09-13T10:02:34.861895+0000","last_clean":"2024-09-13T10:02:34.861895+0000","last_became_active":"2024-09-13T09:23:12.646192+0000","last_became_peered":"2024-09-13T09:23:12.646192+0000","last_unstale":"2024-09-13T10:02:34.861895+0000","last_undegraded":"2024-09-13T10:02:34.861895+0000","last_fullsized":"2024-09-13T10:02:34.861895+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'3","last_scrub_stamp":"2024-09-13T10:02:34.861783+0000","last_deep_scrub":"461'3","last_deep_scrub_stamp":"2024-09-13T10:02:34.861783+0000","last_clean_scrub_stamp":"2024-09-13T10:02:34.861783+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:00.081939+0000","scrub_duration":0.0023050480000000001,"objects_trimmed":0,"snaptrim_duration":0.00035806400000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":491,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:02:44.406220+0000","last_change":"2024-09-13T10:02:44.406220+0000","last_active":"2024-09-13T10:02:44.406220+0000","last_peered":"2024-09-13T10:02:44.406220+0000","last_clean":"2024-09-13T10:02:44.406220+0000","last_became_active":"2024-09-13T09:23:12.645432+0000","last_became_peered":"2024-09-13T09:23:12.645432+0000","last_unstale":"2024-09-13T10:02:44.406220+0000","last_undegraded":"2024-09-13T10:02:44.406220+0000","last_fullsized":"2024-09-13T10:02:44.406220+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:02:44.406070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:02:44.406070+0000","last_clean_scrub_stamp":"2024-09-13T10:02:44.406070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:07:57.799489+0000","scrub_duration":0.0013470419999999999,"objects_trimmed":0,"snaptrim_duration":0.00036873199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":491,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:02:48.392289+0000","last_change":"2024-09-13T10:02:48.392402+0000","last_active":"2024-09-13T10:02:48.392289+0000","last_peered":"2024-09-13T10:02:48.392289+0000","last_clean":"2024-09-13T10:02:48.392289+0000","last_became_active":"2024-09-13T09:23:12.645717+0000","last_became_peered":"2024-09-13T09:23:12.645717+0000","last_unstale":"2024-09-13T10:02:48.392289+0000","last_undegraded":"2024-09-13T10:02:48.392289+0000","last_fullsized":"2024-09-13T10:02:48.392289+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:02:48.392265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:02:48.392265+0000","last_clean_scrub_stamp":"2024-09-13T10:02:48.392265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:46:52.717980+0000","scrub_duration":0.001420992,"objects_trimmed":0,"snaptrim_duration":0.00037606,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"461'2","reported_seq":493,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:02:35.760636+0000","last_change":"2024-09-13T10:02:35.760636+0000","last_active":"2024-09-13T10:02:35.760636+0000","last_peered":"2024-09-13T10:02:35.760636+0000","last_clean":"2024-09-13T10:02:35.760636+0000","last_became_active":"2024-09-13T09:23:12.643941+0000","last_became_peered":"2024-09-13T09:23:12.643941+0000","last_unstale":"2024-09-13T10:02:35.760636+0000","last_undegraded":"2024-09-13T10:02:35.760636+0000","last_fullsized":"2024-09-13T10:02:35.760636+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'2","last_scrub_stamp":"2024-09-13T10:02:35.760513+0000","last_deep_scrub":"461'2","last_deep_scrub_stamp":"2024-09-13T10:02:35.760513+0000","last_clean_scrub_stamp":"2024-09-13T10:02:35.760513+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:55:38.015055+0000","scrub_duration":0.0025736790000000002,"objects_trimmed":0,"snaptrim_duration":0.00041677400000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"461'99","reported_seq":618,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:02:33.772653+0000","last_change":"2024-09-13T10:02:33.772736+0000","last_active":"2024-09-13T10:02:33.772653+0000","last_peered":"2024-09-13T10:02:33.772653+0000","last_clean":"2024-09-13T10:02:33.772653+0000","last_became_active":"2024-09-13T09:23:07.617395+0000","last_became_peered":"2024-09-13T09:23:07.617395+0000","last_unstale":"2024-09-13T10:02:33.772653+0000","last_undegraded":"2024-09-13T10:02:33.772653+0000","last_fullsized":"2024-09-13T10:02:33.772653+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'99","last_scrub_stamp":"2024-09-13T10:02:33.772634+0000","last_deep_scrub":"461'99","last_deep_scrub_stamp":"2024-09-13T10:02:33.772634+0000","last_clean_scrub_stamp":"2024-09-13T10:02:33.772634+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:42.408868+0000","scrub_duration":0.003692191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59912,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738850,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92992,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":89278,"kb_avail":94278848,"statfs":{"total":96636764160,"available":96541540352,"internally_reserved":0,"allocated":3792896,"data_stored":2032126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91421126},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:02:35 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.314,"15min":0.316},"min":{"1min":0.19,"5min":0.19,"15min":0.14799999999999999},"max":{"1min":0.44900000000000001,"5min":0.45400000000000001,"15min":1.1539999999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.373,"5min":0.35499999999999998,"15min":0.34599999999999997},"min":{"1min":0.27300000000000002,"5min":0.189,"15min":0.17599999999999999},"max":{"1min":0.49399999999999999,"5min":0.52200000000000002,"15min":1.0620000000000001},"last":0.318}]},{"osd":1,"last update":"Fri Sep 13 10:02:35 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.312,"15min":0.312},"min":{"1min":0.23899999999999999,"5min":0.218,"15min":0.186},"max":{"1min":0.41999999999999998,"5min":0.48899999999999999,"15min":0.80700000000000005},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.34999999999999998,"15min":0.34699999999999998},"min":{"1min":0.29299999999999998,"5min":0.223,"15min":0.17799999999999999},"max":{"1min":0.45400000000000001,"5min":0.50800000000000001,"15min":1.091},"last":0.35799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738849,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66880,"kb_used_data":3768,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304960,"statfs":{"total":96636764160,"available":96568279040,"internally_reserved":0,"allocated":3858432,"data_stored":2622513,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:02:23 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.314,"15min":0.31},"min":{"1min":0.245,"5min":0.20399999999999999,"15min":0.16},"max":{"1min":0.44400000000000001,"5min":0.44800000000000001,"15min":1.163},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.31,"15min":0.32100000000000001},"min":{"1min":0.26100000000000001,"5min":0.21299999999999999,"15min":0.17999999999999999},"max":{"1min":0.438,"5min":0.45900000000000002,"15min":3.7879999999999998},"last":0.31900000000000001}]},{"osd":2,"last update":"Fri Sep 13 10:02:23 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.32700000000000001,"15min":0.32200000000000001},"min":{"1min":0.27700000000000002,"5min":0.22500000000000001,"15min":0.17000000000000001},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.49199999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.32300000000000001,"15min":0.32600000000000001},"min":{"1min":0.26800000000000002,"5min":0.22,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.63600000000000001},"last":0.32900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738850,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102908,"kb_used_data":3764,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":94268932,"statfs":{"total":96636764160,"available":96531386368,"internally_reserved":0,"allocated":3854336,"data_stored":2621894,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":101513670},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 10:02:25 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.309,"15min":0.308},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.16900000000000001},"max":{"1min":0.41599999999999998,"5min":0.441,"15min":2.911},"last":0.314},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.34000000000000002,"15min":0.32600000000000001},"min":{"1min":0.253,"5min":0.23799999999999999,"15min":0.19500000000000001},"max":{"1min":0.46300000000000002,"5min":0.49299999999999999,"15min":1.044},"last":0.34599999999999997}]},{"osd":2,"last update":"Fri Sep 13 10:02:25 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.312},"min":{"1min":0.24299999999999999,"5min":0.20799999999999999,"15min":0.182},"max":{"1min":0.43099999999999999,"5min":0.47199999999999998,"15min":2.0459999999999998},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.32800000000000001,"15min":0.34000000000000002},"min":{"1min":0.26100000000000001,"5min":0.224,"15min":0.20899999999999999},"max":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":2.4089999999999998},"last":0.38200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T10:02:54.648 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-13T09:23:11.623377+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=9, tm_min=23, tm_sec=11, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=10, tm_min=2, tm_sec=31, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T10:02:54.648 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-13T10:03:14.650 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-13T10:03:14.936 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T10:03:14.937 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-09-13T10:03:14.954 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":1663,"stamp":"2024-09-13T10:03:13.150404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":262780,"kb_used_data":11236,"kb_used_omap":4,"kb_used_meta":251515,"kb_avail":282852740,"statfs":{"total":289910292480,"available":289641205760,"internally_reserved":0,"allocated":11505664,"data_stored":7276533,"data_compressed":59912,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":257551698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002424"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":490,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:02:49.920755+0000","last_change":"2024-09-13T10:02:49.920868+0000","last_active":"2024-09-13T10:02:49.920755+0000","last_peered":"2024-09-13T10:02:49.920755+0000","last_clean":"2024-09-13T10:02:49.920755+0000","last_became_active":"2024-09-13T09:23:12.644939+0000","last_became_peered":"2024-09-13T09:23:12.644939+0000","last_unstale":"2024-09-13T10:02:49.920755+0000","last_undegraded":"2024-09-13T10:02:49.920755+0000","last_fullsized":"2024-09-13T10:02:49.920755+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:02:49.920732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:02:49.920732+0000","last_clean_scrub_stamp":"2024-09-13T10:02:49.920732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:14:07.062623+0000","scrub_duration":0.0013463819999999999,"objects_trimmed":0,"snaptrim_duration":0.00075243300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":490,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:02:42.860198+0000","last_change":"2024-09-13T10:02:42.860319+0000","last_active":"2024-09-13T10:02:42.860198+0000","last_peered":"2024-09-13T10:02:42.860198+0000","last_clean":"2024-09-13T10:02:42.860198+0000","last_became_active":"2024-09-13T09:23:12.646762+0000","last_became_peered":"2024-09-13T09:23:12.646762+0000","last_unstale":"2024-09-13T10:02:42.860198+0000","last_undegraded":"2024-09-13T10:02:42.860198+0000","last_fullsized":"2024-09-13T10:02:42.860198+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:02:42.860176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:02:42.860176+0000","last_clean_scrub_stamp":"2024-09-13T10:02:42.860176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:28.485888+0000","scrub_duration":0.0013819220000000001,"objects_trimmed":0,"snaptrim_duration":0.00051643700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":491,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:02:37.830162+0000","last_change":"2024-09-13T10:02:37.830162+0000","last_active":"2024-09-13T10:02:37.830162+0000","last_peered":"2024-09-13T10:02:37.830162+0000","last_clean":"2024-09-13T10:02:37.830162+0000","last_became_active":"2024-09-13T09:23:12.644747+0000","last_became_peered":"2024-09-13T09:23:12.644747+0000","last_unstale":"2024-09-13T10:02:37.830162+0000","last_undegraded":"2024-09-13T10:02:37.830162+0000","last_fullsized":"2024-09-13T10:02:37.830162+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:02:37.830020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:02:37.830020+0000","last_clean_scrub_stamp":"2024-09-13T10:02:37.830020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:41.135864+0000","scrub_duration":0.001460469,"objects_trimmed":0,"snaptrim_duration":0.00044772799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":491,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:02:43.842275+0000","last_change":"2024-09-13T10:02:43.842275+0000","last_active":"2024-09-13T10:02:43.842275+0000","last_peered":"2024-09-13T10:02:43.842275+0000","last_clean":"2024-09-13T10:02:43.842275+0000","last_became_active":"2024-09-13T09:23:12.644527+0000","last_became_peered":"2024-09-13T09:23:12.644527+0000","last_unstale":"2024-09-13T10:02:43.842275+0000","last_undegraded":"2024-09-13T10:02:43.842275+0000","last_fullsized":"2024-09-13T10:02:43.842275+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:02:43.842073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:02:43.842073+0000","last_clean_scrub_stamp":"2024-09-13T10:02:43.842073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:03.453693+0000","scrub_duration":0.0019788430000000001,"objects_trimmed":0,"snaptrim_duration":0.00071815500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"461'3","reported_seq":495,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:02:34.861895+0000","last_change":"2024-09-13T10:02:34.861895+0000","last_active":"2024-09-13T10:02:34.861895+0000","last_peered":"2024-09-13T10:02:34.861895+0000","last_clean":"2024-09-13T10:02:34.861895+0000","last_became_active":"2024-09-13T09:23:12.646192+0000","last_became_peered":"2024-09-13T09:23:12.646192+0000","last_unstale":"2024-09-13T10:02:34.861895+0000","last_undegraded":"2024-09-13T10:02:34.861895+0000","last_fullsized":"2024-09-13T10:02:34.861895+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'3","last_scrub_stamp":"2024-09-13T10:02:34.861783+0000","last_deep_scrub":"461'3","last_deep_scrub_stamp":"2024-09-13T10:02:34.861783+0000","last_clean_scrub_stamp":"2024-09-13T10:02:34.861783+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:00.081939+0000","scrub_duration":0.0023050480000000001,"objects_trimmed":0,"snaptrim_duration":0.00035806400000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":491,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:02:44.406220+0000","last_change":"2024-09-13T10:02:44.406220+0000","last_active":"2024-09-13T10:02:44.406220+0000","last_peered":"2024-09-13T10:02:44.406220+0000","last_clean":"2024-09-13T10:02:44.406220+0000","last_became_active":"2024-09-13T09:23:12.645432+0000","last_became_peered":"2024-09-13T09:23:12.645432+0000","last_unstale":"2024-09-13T10:02:44.406220+0000","last_undegraded":"2024-09-13T10:02:44.406220+0000","last_fullsized":"2024-09-13T10:02:44.406220+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:02:44.406070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:02:44.406070+0000","last_clean_scrub_stamp":"2024-09-13T10:02:44.406070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:07:57.799489+0000","scrub_duration":0.0013470419999999999,"objects_trimmed":0,"snaptrim_duration":0.00036873199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":491,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:02:48.392289+0000","last_change":"2024-09-13T10:02:48.392402+0000","last_active":"2024-09-13T10:02:48.392289+0000","last_peered":"2024-09-13T10:02:48.392289+0000","last_clean":"2024-09-13T10:02:48.392289+0000","last_became_active":"2024-09-13T09:23:12.645717+0000","last_became_peered":"2024-09-13T09:23:12.645717+0000","last_unstale":"2024-09-13T10:02:48.392289+0000","last_undegraded":"2024-09-13T10:02:48.392289+0000","last_fullsized":"2024-09-13T10:02:48.392289+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T10:02:48.392265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T10:02:48.392265+0000","last_clean_scrub_stamp":"2024-09-13T10:02:48.392265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:46:52.717980+0000","scrub_duration":0.001420992,"objects_trimmed":0,"snaptrim_duration":0.00037606,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"461'2","reported_seq":493,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:02:35.760636+0000","last_change":"2024-09-13T10:02:35.760636+0000","last_active":"2024-09-13T10:02:35.760636+0000","last_peered":"2024-09-13T10:02:35.760636+0000","last_clean":"2024-09-13T10:02:35.760636+0000","last_became_active":"2024-09-13T09:23:12.643941+0000","last_became_peered":"2024-09-13T09:23:12.643941+0000","last_unstale":"2024-09-13T10:02:35.760636+0000","last_undegraded":"2024-09-13T10:02:35.760636+0000","last_fullsized":"2024-09-13T10:02:35.760636+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'2","last_scrub_stamp":"2024-09-13T10:02:35.760513+0000","last_deep_scrub":"461'2","last_deep_scrub_stamp":"2024-09-13T10:02:35.760513+0000","last_clean_scrub_stamp":"2024-09-13T10:02:35.760513+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:55:38.015055+0000","scrub_duration":0.0025736790000000002,"objects_trimmed":0,"snaptrim_duration":0.00041677400000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"461'99","reported_seq":618,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-13T10:02:33.772653+0000","last_change":"2024-09-13T10:02:33.772736+0000","last_active":"2024-09-13T10:02:33.772653+0000","last_peered":"2024-09-13T10:02:33.772653+0000","last_clean":"2024-09-13T10:02:33.772653+0000","last_became_active":"2024-09-13T09:23:07.617395+0000","last_became_peered":"2024-09-13T09:23:07.617395+0000","last_unstale":"2024-09-13T10:02:33.772653+0000","last_undegraded":"2024-09-13T10:02:33.772653+0000","last_fullsized":"2024-09-13T10:02:33.772653+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'99","last_scrub_stamp":"2024-09-13T10:02:33.772634+0000","last_deep_scrub":"461'99","last_deep_scrub_stamp":"2024-09-13T10:02:33.772634+0000","last_clean_scrub_stamp":"2024-09-13T10:02:33.772634+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:42.408868+0000","scrub_duration":0.003692191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59912,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92992,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":89278,"kb_avail":94278848,"statfs":{"total":96636764160,"available":96541540352,"internally_reserved":0,"allocated":3792896,"data_stored":2032126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91421126},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:02:35 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.314,"15min":0.316},"min":{"1min":0.19,"5min":0.19,"15min":0.14799999999999999},"max":{"1min":0.44900000000000001,"5min":0.45400000000000001,"15min":1.1539999999999999},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.35499999999999998,"15min":0.34599999999999997},"min":{"1min":0.27300000000000002,"5min":0.189,"15min":0.17599999999999999},"max":{"1min":0.49399999999999999,"5min":0.52200000000000002,"15min":1.0620000000000001},"last":0.309}]},{"osd":1,"last update":"Fri Sep 13 10:02:35 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.312,"15min":0.312},"min":{"1min":0.23899999999999999,"5min":0.218,"15min":0.186},"max":{"1min":0.41999999999999998,"5min":0.48899999999999999,"15min":0.80700000000000005},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.34999999999999998,"15min":0.34699999999999998},"min":{"1min":0.29299999999999998,"5min":0.223,"15min":0.17799999999999999},"max":{"1min":0.45400000000000001,"5min":0.50800000000000001,"15min":1.091},"last":0.33800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738853,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66880,"kb_used_data":3768,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304960,"statfs":{"total":96636764160,"available":96568279040,"internally_reserved":0,"allocated":3858432,"data_stored":2622513,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 10:02:23 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.314,"15min":0.31},"min":{"1min":0.245,"5min":0.20399999999999999,"15min":0.16},"max":{"1min":0.44400000000000001,"5min":0.44800000000000001,"15min":1.163},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.31,"15min":0.32100000000000001},"min":{"1min":0.26100000000000001,"5min":0.21299999999999999,"15min":0.17999999999999999},"max":{"1min":0.438,"5min":0.45900000000000002,"15min":3.7879999999999998},"last":0.32400000000000001}]},{"osd":2,"last update":"Fri Sep 13 10:02:23 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.32700000000000001,"15min":0.32200000000000001},"min":{"1min":0.27700000000000002,"5min":0.22500000000000001,"15min":0.17000000000000001},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.49199999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.32300000000000001,"15min":0.32600000000000001},"min":{"1min":0.26800000000000002,"5min":0.22,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.63600000000000001},"last":0.26700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738854,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102908,"kb_used_data":3764,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":94268932,"statfs":{"total":96636764160,"available":96531386368,"internally_reserved":0,"allocated":3854336,"data_stored":2621894,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":101513670},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 10:02:25 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.309,"15min":0.308},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.16900000000000001},"max":{"1min":0.41599999999999998,"5min":0.441,"15min":2.911},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.34000000000000002,"15min":0.32600000000000001},"min":{"1min":0.253,"5min":0.23799999999999999,"15min":0.19500000000000001},"max":{"1min":0.46300000000000002,"5min":0.49299999999999999,"15min":1.044},"last":0.29199999999999998}]},{"osd":2,"last update":"Fri Sep 13 10:02:25 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.312},"min":{"1min":0.24299999999999999,"5min":0.20799999999999999,"15min":0.182},"max":{"1min":0.43099999999999999,"5min":0.47199999999999998,"15min":2.0459999999999998},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.32800000000000001,"15min":0.34000000000000002},"min":{"1min":0.26100000000000001,"5min":0.224,"15min":0.20899999999999999},"max":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":2.4089999999999998},"last":0.246}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T10:03:14.956 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-09-13T10:03:15.246 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-13T10:03:15.246 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-13T10:03:15.246 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-09-13T10:03:15.246 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T10:03:15.291 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-09-13T10:03:15.291 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-09-13T10:03:15.291 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T10:03:15.329 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-09-13T10:03:15.329 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-09-13T10:03:15.329 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T10:03:15.410 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-09-13T10:03:15.411 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-13T10:03:15.411 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-09-13T10:03:15.411 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T10:03:15.477 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-09-13T10:03:15.477 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-13T10:03:15.477 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-09-13T10:03:15.477 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T10:03:15.530 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-09-13T10:03:15.530 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-13T10:03:15.530 DEBUG:teuthology.orchestra.run.smithi122:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-13T10:03:15.598 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T10:03:15.598 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-09-13T10:03:15.805 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T10:03:15.805 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-09-13T10:03:15.910 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T10:03:15.910 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-09-13T10:03:15.968 INFO:tasks.ceph:Archiving mon data... 2024-09-13T10:03:15.968 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7901034/data/cluster1.mon.a.tgz 2024-09-13T10:03:15.969 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-09-13T10:03:15.983 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.GYTltihMZi 2024-09-13T10:03:15.984 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.GYTltihMZi 2024-09-13T10:03:16.147 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.GYTltihMZi 2024-09-13T10:03:16.231 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.GYTltihMZi is 401KB 2024-09-13T10:03:16.329 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.GYTltihMZi 2024-09-13T10:03:16.344 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-13T10:03:16.344 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T10:03:16.385 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T10:03:16.416 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-13T10:03:16.416 DEBUG:teuthology.orchestra.run.smithi122:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-13T10:03:16.450 DEBUG:teuthology.orchestra.run.smithi190:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-13T10:03:16.474 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-13T10:03:16.532 INFO:teuthology.orchestra.run.smithi190.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-13T10:03:16.537 INFO:tasks.ceph:Archiving crash dumps... 2024-09-13T10:03:16.539 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7901034/remote/smithi122/crash 2024-09-13T10:03:16.540 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T10:03:16.565 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7901034/remote/smithi190/crash 2024-09-13T10:03:16.566 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T10:03:16.594 INFO:tasks.ceph:Compressing logs... 2024-09-13T10:03:16.595 DEBUG:teuthology.orchestra.run.smithi122:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T10:03:16.610 DEBUG:teuthology.orchestra.run.smithi190:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T10:07:41.145 INFO:tasks.ceph:Archiving logs... 2024-09-13T10:07:41.145 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7901034/remote/smithi122/log 2024-09-13T10:07:41.146 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T10:07:44.193 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7901034/remote/smithi190/log 2024-09-13T10:07:44.194 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T10:08:16.889 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T10:08:16.902 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T10:08:16.902 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T10:08:16.906 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T10:08:17.048 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T10:08:17.060 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T10:08:17.060 DEBUG:teuthology.orchestra.run.smithi122:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T10:08:17.063 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T10:08:17.078 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-09-13T10:08:17.081 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-09-13T10:08:17.083 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T10:08:17.083 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-09-13T10:08:17.083 INFO:teuthology.orchestra.run.smithi122.stdout:^* hv01.front.sepia.ceph.com 3 7 377 79 -215us[ -217us] +/- 43ms 2024-09-13T10:08:17.083 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 80 +277us[ +274us] +/- 56ms 2024-09-13T10:08:17.084 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 213 +20us[ +39us] +/- 68ms 2024-09-13T10:08:17.084 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T10:08:17.121 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T10:08:17.121 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-09-13T10:08:17.122 INFO:teuthology.orchestra.run.smithi190.stdout:^* hv01.front.sepia.ceph.com 3 6 377 23 -240us[ -244us] +/- 44ms 2024-09-13T10:08:17.122 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 17 +280us[ +280us] +/- 57ms 2024-09-13T10:08:17.122 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 +2820ns[+2820ns] +/- 71ms 2024-09-13T10:08:17.122 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T10:08:17.123 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T10:08:17.135 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T10:08:17.135 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T10:08:17.159 DEBUG:teuthology.orchestra.run.smithi122:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T10:08:17.226 DEBUG:teuthology.orchestra.run.smithi190:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T10:08:17.337 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-13T10:08:17.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T10:08:17.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-13T10:08:17.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T10:08:17.397 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T10:08:17.409 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T10:08:17.459 INFO:teuthology.task.internal:Duration was 3091.793534 seconds 2024-09-13T10:08:17.460 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T10:08:17.543 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T10:08:17.543 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T10:08:17.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T10:08:17.586 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T10:08:17.617 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T10:08:18.093 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T10:08:18.093 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T10:08:18.093 DEBUG:teuthology.orchestra.run.smithi122:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T10:08:18.170 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2024-09-13T10:08:18.171 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T10:08:18.194 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T10:08:18.195 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T10:08:18.215 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T10:08:18.267 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T10:08:18.267 DEBUG:teuthology.orchestra.run.smithi122:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T10:08:18.284 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T10:08:18.427 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T10:08:18.533 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T10:08:18.534 DEBUG:teuthology.orchestra.run.smithi122:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T10:08:18.558 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T10:08:18.584 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T10:08:18.596 DEBUG:teuthology.orchestra.run.smithi122:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T10:08:18.602 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T10:08:18.622 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = core 2024-09-13T10:08:18.657 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2024-09-13T10:08:18.670 DEBUG:teuthology.orchestra.run.smithi122:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T10:08:18.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T10:08:18.718 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T10:08:18.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T10:08:18.732 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T10:08:18.747 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T10:08:18.748 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7901034/remote/smithi122 2024-09-13T10:08:18.748 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T10:08:18.793 DEBUG:teuthology.misc:Transferring archived files from smithi190:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7901034/remote/smithi190 2024-09-13T10:08:18.793 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T10:08:18.826 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T10:08:18.827 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T10:08:18.829 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T10:08:18.877 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T10:08:18.893 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T10:08:18.894 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T10:08:18.927 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T10:08:18.927 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T10:08:18.929 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T10:08:18.942 INFO:teuthology.orchestra.run.smithi122.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 10:08 /home/ubuntu/cephtest 2024-09-13T10:08:18.944 INFO:teuthology.orchestra.run.smithi190.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 10:08 /home/ubuntu/cephtest 2024-09-13T10:08:18.945 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T10:08:18.968 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T10:08:19.080 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} duration: 3091.7935342788696 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-13T10:08:19.080 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T10:08:19.162 INFO:teuthology.run:pass